Methods and apparatus for suggesting and/or associating tags corresponding to identified image content and/or storing said image content in association with tags to facilitate retrieval and use

ABSTRACT

Methods and apparatus for using one or more feature recognition system(s) to identify features in a scanned image and to associate tags with identified features are described. Probabilities of a feature being present are taken into consideration in some embodiments. In various embodiments once a tag has been determined as corresponding to a feature which has been confirmed as being in the scanned image synonyms for the tag word are identified and also associated with the scanned image. By using results of multiple automated feature recognition systems and generating overall probabilities that a feature is present in an image more reliable tagging can be implemented in an automated manner than in system which rely on a single AI system without the need for extensive operator input with respect to identification of all features in the image which are to be tagged.

RELATED APPLICATIONS

The present application claims the benefit of the filing date of U.S.Provisional Patent Application Ser. No. 62/813,091 which was filed Mar.3, 2019, U.S. Provisional Patent Application Ser. No. 62/813,134 whichwas filed Mar. 3, 2019, and is a continuation in part of U.S. patentapplication Ser. No. 16/799,795 which was filed on Feb. 24, 2020 andwhich claims benefit of U.S. Provisional Patent Application Ser. No.62/809,716 which was filed Feb. 24, 2019 and wherein each of thepreceding applications are expressly incorporated by reference in theirentirety.

FIELD OF THE INVENTION

The invention relates to feature detection and image tagging and, moreparticularly, to methods and apparatus for determining features in oneor more scanned images, associating tags with scanned images and/orusing tagged images.

BACKGROUND

Images may be, and often are, scanned for a variety of purposes. In manycases the scanned images are photos. The photos may be personal photosto be archived for personal reasons or photos of a more general naturescanned for other reasons.

While the scanning of photos can be a quick and simple process, oncedigitized the photo can be stored with numerous other photos. To lateridentify a scanned photo, one may need to search through hundreds ofphotos to find one the user is looking for.

To facilitate later identification and retrieval, e.g., using a wordbased search, a tag indicating a feature of the photo may be associatedwith the photo. Tagging of photos can be a labor intensive and timeconsuming processes since it often involves a human looking at a photoand associating one or more tags, e.g., words, with the photo, by typingin the word to be used as a tag.

While having a human manually tag a photo based on features the userobserves and/or considers significant in the photo is one approach totagging photos, such a fully human based approach suffers variousproblems in addition to being labor intensive. Among the problems isthat in having a single human tag a scanned image such as a photo, theuser may miss one or more features that someone else might consider ofinterest. In addition, the vocabulary of the user doing the tagging mayaffect the word used for the tag or tags being used for a feature. Thusa person later seeking to find a photo may miss a photo because it wastagged using a word which is different from the word the person seekingto find the photo uses for a feature which was identified by the persondoing the tagging but labeled with a different word.

Various commercial systems have attempted to address some of theproblems associated with identifying and tagging features in a scannedimage. Such systems are often based on Artificial Intelligence (AI)being used to do pattern recognition on a scanned image and thenproviding a list of words, e.g., tags, corresponding to identifiedfeatures. In addition to each feature, by providing the tag, e.g., word,corresponding to an identified feature, such systems often provide aprobability that the features to which the tag corresponds is in factpresent in the image. Unfortunately the reliability of AI systems toidentify features is often less reliable than is desirable and oftenvaries from one AI system to another for a variety of reasons includingthe training data used to train the particular system and/or the featureidentification algorithm or neural network used for identification.

Given the less than desirable reliability of individual AI systems atfeature detection, for many applications individual AI systems remaininsufficiently reliable for automating a feature identification andtagging process.

In view of the above it should be appreciated that there are needs formethods and apparatus for improving feature detection and/or tagging ofscanned images for a variety of purposes including, but not limited to,photo achieving applications.

SUMMARY OF THE INVENTION

Methods and apparatus for facilitating the use of automatic featuredetection in image processing are described. The methods and apparatusare well suited for use in systems used for detecting features in imagesand/or tagging scanned image content to indicate features included inthe images.

In various embodiments a scanned image is supplied to multipleartificial intelligence (AI) systems which provide feature detectionservices and which provide a tag corresponding to an identified featurewith an indication of the probability that the feature is present in theprocessed image.

To overcome the unreliability of individual AI systems with regard toaccurately detecting features in an image, Bayesian inference techniquesare used to process the output of multiple AI systems whichindependently process the scanned image, e.g., digitized photo.

A Bayesian inference is a method of statistical inference in whichBayes' theorem is used to update the probability for a hypothesis asmore evidence or information becomes available. In the case of featuredetection is the hypothesis. In Bayesian statistics, the posteriorprobability of a random event or an uncertain proposition is theconditional probability that is assigned after the relevant evidence orbackground is taken into account.

In various embodiments the detection, by one AI system, of a feature andthe corresponding probability that the feature is present in an image isused as a prior event input to a probability function that also receivesa detection confidence value generated by another AI system thatdetected the same feature in an image. By using multiple different AIsystems and processing their confidence of detection of a particularfeature, a more reliable confidence score can be generated than using asingle AI system since different AI systems suffer from differentdeficiencies. The resulting confidence score in the form of a posteriordensity value determined using the output of multiple AI systems can be,and sometimes is, then compared to a threshold to determine if a featureis present.

Accordingly, various features relate to using multiple AI systems andconfidence results in combination with Bayesian inference analysis todetermine if one or more features are present in an attempt to increasethe overall reliability of feature detection and thus facilitateautomation of at least some tagging operations by reaching a level ofreliability that allows for at least some tags to automatically beassociated with a scanned image.

In some embodiments human confirmation of some features prior toassociating a corresponding tag with an image is supported. Featuressubject to human confirmation may, and sometimes do, have an overalldetection reliability score below a level used for automatic tagassociation. Since much of the tagging can be automated and humaninvolvement with individual features is reduced to a subset of featuresidentified by AI processing of the image, the burden of initial featureidentification is removed from the human system operator and the humanoperator is limited in the number of features they need toreview/confirm as part of the tagging process.

Other features relate to addressing the problem of what terms should beused for tags and attempt to address the potentially limited or biasedvocabulary of the human operator of the system.

In various embodiments once a feature has been identified and a decisionmade to associate a particular word tag with the image having theidentified feature, synonyms for the tag are identified through anautomated synonym look up operation. The tag and the synonyms are thenassociated with the image, e.g., automatically or in some cases after anoperator confirms that the synonym should be used as a tag. Byperforming a synonym lookup operation and association, the risk of an AIsystems limited terminology or an operator's limited vocabulary failingto use an appropriate tag once a feature has been identified is reducedas compared to systems which do not add such synonym's.

Various additional features and advantages of the present invention arediscussed in the detailed description which follows.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary communications system in which the homeappliance and secure server of the present invention may be used.

FIG. 2 illustrates an exemplary home appliance implemented in accordancewith the invention which may be used in the system of FIG. 1.

FIG. 3 is a more detailed illustration of the exemplary home applianceshown in FIG. 2

FIG. 4 is another exemplary communications system implemented inaccordance with the present invention which includes more detail than isincluded in FIG. 1.

FIG. 5 illustrates a home appliance with a detachable display andscanner implemented in accordance with one exemplary embodiment.

FIG. 6 shows how the components of the system shown in FIG. 5 can bedetached from one another and used as they communicate wirelessly witheach other.

FIG. 7 illustrates a household control panel which may be accessed viathe home appliance of the present invention.

FIG. 8 illustrates a family command and control center screen which maybe accessed and used to communicate household information, accesshousehold calendars, etc.

FIG. 9 illustrates a secure server which may be used as the secureserver of the system shown in FIGS. 1 and 4, and which interacts withthe home appliance in a secure manner to provide a wide range ofservices to a household and the various members of a household.

FIG. 10 illustrates how asset ownership and/or access rights may be andsometimes are automatically changed in response to one or more lifeevents that can be specified by a user and/or determined or verifiedbased on one or more scanned documents.

FIG. 11A is a first part of the flow chart of FIG. 11.

FIG. 11B is a second part of the flow chart of FIG. 11.

FIG. 11C is a third part of the flow chart of FIG. 11.

FIG. 11 shows how FIGS. 11A, 11B and 11C can be combined to form a flowchart showing steps relating to asset storage, ownership and/or accesscontrol which are used in various embodiments of the present inventionto manage contents of a digital data store that may be and sometimes isa remote cloud based data store including family information.

FIG. 12 shows the steps of a method of processing a physical image inthe form of a photo or document to digitize the image and associate tagswith the image for storage and to facilitate possible future word basedimage searching and image retrieval.

FIG. 13 shows the steps of a method of using information from multipleartificial intelligence systems to identify one or more features in ascanned image and determine an overall confidence level for one or moreindividual features based on information provided by multiple artificialintelligence systems to which Bayesian inference analysis is applied.

FIG. 14 shows the steps of a method of determine whether a tag or tagsshould be associated with an image based on results of one or moreartificial intelligence feature recognition operations and/or operatorinput.

FIG. 15 illustrate an exemplary artificial intelligence (AI) imagefeature identifier device, e.g. server, in accordance with an exemplaryembodiment.

FIG. 16A is a first part of an exemplary assembly of components whichmay be included in a device in accordance with an exemplary embodiment.

FIG. 16B is a second part of an exemplary assembly of components whichmay be included in a device in accordance with an exemplary embodiment.

FIG. 16 comprises the combination of FIG. 16A and FIG. 16B.

DETAILED DESCRIPTION

FIG. 1 illustrates an exemplary system 100 in which one or moreappliances, e.g., a home appliances 108, 111 of the invention may beused to input data which can be added to a digital data store. The homeappliances 108, 110 can scan images, e.g., documents or photos, and sendthe scanned images to one or more of the Artificial Intelligence (AI)systems 97, 98, 99 via communications network 104 and/or the secureserver 102. In some embodiments the AI systems 97, 98, 99 identifyfeatures in the scanned images and report back to the home appliance108, 111 from which a scanned image was received for processing. Theinformation provided back to the appliance includes a list of identifiedfeatures, one or more tags, e.g., words, corresponding to eachidentified feature and a separate confidence probability for eachidentified feature indicating the probability that the identifiedfeature is in the scanned image. The different AI systems 97, 98, 99 maybe, and sometimes are, different commercial systems running different AIprocesses and/or were trained different image training sets. The AIsystems 97, 98, 99 may, and sometimes will, return different featuredetection results. As will be discussed below, an appliance 108, 111can, and sometimes does, processes the results provided by different AIsystems 97, 98, 99 to determine what features were consistentlydetermined to be present in a scanned image and what tag or tags, e.g.,words, to associate with the scanned image when it is stored in memoryto thereby facilitate word searching of images and/or image retrieval.Each AI device includes a processor, memory and an input/outputinterface. The processor controls the AI device to perform featuredetection and return feature detection reports including suggested tagsand an indication of a confidence level for each detected featurereported back to a device which supplied a scanned image.

The system 100 includes AI devices 97, 98, 99, a secure server 102 of aprovider of a service, e.g., a home information and/or communicationsmanagement service, a health care service provider system 114, anindividual system 112 to which photos and/or other data is to bedistributed and a plurality of customer premise locations 106, 110,e.g., household locations, each of which includes an home appliance 108,111 implemented in accordance with the present invention. Variouselements of the system 100 are coupled together via a communicationsnetwork 104 which may be the Internet, a public telephone network, or acombination of one or more networks. While the communications network104 may be a public network, communications links over the network may,and in various embodiments are, secured using encryption and/or othertechniques. The communications links between the home appliances 108,111 may be Virtual Private Network (VPN) communications links. Access toother systems, e.g., the health care service provider system 114,Internet search provider systems, E-mail systems, etc. via the homeappliance may be protected by a firewall included in or managed by theservice provider's secure server 102. The secure server 102 supportsvarious functions and can interact with the home appliances 108, 111 ofthe present invention to provide a wide variety of services, informationmanagement functions, device information synchronization functions,communication service management functions, etc. in a secure manner. Theservices/functions supported by the secure server 102 include one ormore of the following: i) media (e.g., image/film/document) archiving,documenting and distribution functions, ii) video phone service; iii)conferencing and on-line collaboration functions, e.g., allowing forsimultaneous document viewing of documents or images by users each ofwhich has an appliance of the present invention; iv) monitoringfunctions such as health monitoring functions; v) communications servicemanagement functions, vi) device data synchronization functions; andvii) household bulletin board functions, etc.

FIG. 2 illustrates an exemplary table top household appliance 200implemented in accordance with the invention. The household appliance200 includes a housing (201, 205 in combination) which is formed by anupper housing portion 201 and a lower housing portion 205 coupledtogether by a hinge 203. Mounted to the housing, and thus integraltherewith, is a camera 206, display screen 204, a memory card slot 207,photo/document scanner 212, microphone 210, speaker 214, optionalkeyboard 216, arrow keys 220, select key 218 and various interfaces 222,224. The display screen 204 may be, and in some embodiments is, a colortouch screen. In various touch screen embodiments, keyboard 216 may beomitted. The interfaces 222, 224 may be wired interfaces, wirelessinterfaces and/or a combination of wired and wireless interfaces. In thecase of wireless interfaces, the interface 222 or 224 may include bothreceiver and transmitter circuitry for receiving and transmittingsignals. Internal to the appliance 200, and thus not visible in the FIG.2 illustration, is a processor and memory. The processor controlsoperation of the device under direction of one or more modules, e.g.,routines, stored in memory. The memory may also be used to storedocument images, photo images, etc. However, in order to keepimplementation costs low, in some embodiments the apparatus 200 includesa relatively small amount of memory with the appliance relying onnetwork connectivity and network storage for data intensive functionssuch as storing large photo albums and/or large numbers of documents.Additional storage can be added to the apparatus by inserting a memorycard, e.g., SD, XD or other type of memory card, in card slot 207 or byattaching an external media device 242, e.g., a USB hard disc drive, toone of the I/O interfaces 222 or 224. The table top appliance 200 can,and sometimes does, perform optical character recognition and thenperform various asset ownership and/or asset management/access controlfunctions based on the scanned or user provided input. In otherembodiments the device 200 communicates scanned documents and/or userinput.

Various peripheral devices, e.g., a health monitoring apparatus 240, maybe added to, e.g., coupled to, the appliance 200, to supplement thestand alone appliance's capabilities. Various peripheral devices used insome embodiments include one or more of the following: a media readersuch as one or more of the following: a slide reader, a cassetterecorder reader (audio or video), a floppy disk reader, a 78 recordplayback drive, a reel to real tape reader, a film reader. The variousreaders digitize analog input which is then processed and stored indigital form, e.g., in one or more files and/or communicated via acommunications network, e.g., to a network server for storage and/ordistribution.

In some embodiments, where healthcare is supported, the appliance 200 iscoupled to a monitoring apparatus for monitoring one or more sensorinputs. The sensor inputs may include one or more human health monitors,motion monitors and/or a variety of environmental condition monitorssuch as security sensors, temperature sensors, etc. A blood pressuremonitor, glucose level monitor, heart rate monitor, and blood oxygenlevel monitor are amount the various sensors and monitors which are usedand supported in some embodiments. In some embodiments interface 222 iscoupled to a sensor network from which the appliance 200 receivessignals used to detect at least one of movement and proximity of aliving being.

In some embodiments the appliance 200 is configured to generate an alarmand automatically initiate an audio/video conference in the event motionis not detected at predetermined intervals or during a monitoring perioddetermined according to a programmed schedule. For example, the device200 may be set to detect motion in the morning and, failing to detectthe motion may set off an alarm based on the concern that the residentmay have been unable to get up or make to the room in which theappliance 200 is located to take medication at a scheduled time. Thedevice 200 may be set to monitor for audio alarms, e.g., a personyelling for help, or for an alarm signal which may be transmitted to thedevice 200 by an item worn on an individual expected to use theappliance 200. In the event of an alarm condition, the appliance 200 mayinitiate a video conference call in which audio as well as video may bemonitored and an individual at the customer premise at which theappliance 200 is located may communicate with an individual at anotherlocation, e.g., monitoring location, without having to get up or move,through the automatic use of the microphone (mic) 210 and speaker 214.

In the FIG. 2 example, display 204 illustrates an exemplary opening menu205. The appliance 200 may be preconfigured to display the opening menuupon power up. The menu 205 lists various selection options including afirst option corresponding to a video conference function, a secondoption corresponding to various photo and/or document functions, a thirdoption relating to sensor and/or health monitoring functions and afourth option relating to system configuration. Each of the functionsmay be highlighted by moving around on the screen using arrow keys 220and by pressing select key 218 to indicate a user selection.Alternatively, in touch screen embodiments, a user of the appliance mayselect an option by touching the portion of the display screen 204 onwhich the desired option is displayed.

Upon selection of the video conference option, a user is presented withan additional screen that allows a user to initiate or join a videoconference, e.g., by entering an identifier corresponding to a party orparties which are to be contacted or by accepting a received invitationto join a video conference initiated by another party, e.g., anindividual using another appliance 200 of the present invention.

Upon selection of the photo/document functions option, a user ispresented with a menu that allows the user to maker further selectionsrelating to the scanning of photos or documents including the optionalrecording of voice commentary and an image of the individual providingthe commentary, e.g., as part of creating an electronic collection ofphotos or documents, e.g., an album, which can be uploaded to a server,stored on the server and/or distributed by the server. Given the limitedmemory on the appliance 200, in some embodiments individual images anddocuments, along with any commentary or image of the person providingthe commentary, are uploaded to a server via a network interface withthe server then storing the multiple images and compiling them intoalbums in accordance with input received from the appliance 200.

Upon selection of sensor/health monitor functions, the user is providedwith additional options and supported actions relating to the sensorsand/or health monitoring peripheral device 240 coupled to the appliance200. The appliance 200 supports automatic detection and configuration ofperipheral devices. Accordingly, as a user adds or removes peripheraldevices the options available upon selection of the sensor/healthmonitoring functions option will change depending on the sensors/healthmonitoring apparatus present at a particular point in time.

Upon selection of the configure system option, a user is provided withvarious configuration options, e.g. display and other setting options. Auser may provide a security key, e.g., a Wired Equivalent Privacy (WEP)key, required to obtain wireless connectivity to a local network viasetting entry options presented after the configure system option isselected. While a user may configure the appliance 200 manually, thedevice can also be configured remotely, e.g., by a server in the networkto which the apparatus 200 is connected. A telephone interface andpre-programmed telephone number may be included in the apparatus 200 forobtaining configuration information via the public telephone network.However, where wireless or other Internet connectivity is supported, theappliance may connect via such a connection to a network based server toobtain additional or updated configuration information or to downloadnew application software, e.g., relating to one or more peripheraldevices which may be added to the apparatus 200.

FIG. 3 illustrates, in block diagram form, a customer/home managementappliance 300 implemented in accordance with one exemplary embodiment ofthe present invention. The appliance 200 may include the same elementsas the appliance 300 shown in FIG. 3. The appliance 300 includes ainput/output interface 304, processor 306, assembly of components 352,e.g. assembly of hardware components, e.g., circuits, display 308,optional keypad 310, speaker 313, microphone 314, camera 316, scanner318, peripheral device I/O 320, memory 302, optional local storagedevice 322 and a local sensor I/O 324 coupled together by a bus 321which allow the various components to exchange data and other signals.The various components are securely, e.g., permanently, mounted to ahousing and thus are integral to the appliance 300. The housing may be asingle fixed assembly or a multi-part housing such as the multiparthousing including upper and lower parts 201, 205 shown in the FIG. 2example.

The I/O module 304 severs as a network interface and couples theappliance 300 to a communications network, e.g., the Internet and/orpublic telephone network as indicated by communications link 350. TheI/O 304 may be implemented as a wired and/or wireless interface. In someembodiments, the I/O interface 304 includes a wired receiver (RX) 356and a wired transmitter (TX) 358. In some embodiments, the I/O interface304 includes a wireless receiver (RX) 360 coupled to receive antenna 364and a wireless transmitter (TX) 362 coupled to transmit antenna 366. Inone embodiment the I/O 304 includes an Ethernet port for connection towired networks, a telephone jack for connection to standard telephonenetworks and also a wireless interface, e.g., a WiFi interface forconnecting to a wireless network. Other types of network interfaces canbe supported such a conventional cell phone interfaces thereby allowingthe customer appliance to communicate with other devices and networkservers using any one of a wide variety of communications networks.

Processor 306, e.g., a CPU, controls operation of the customer appliance300 in accordance with one or more control routines stored in memory302. The processor 306 controls, among other things, the presentation ofmenus and prompts on display 308 and the disposition of scanned imagesand/or other files in accordance with user input, e.g., menu selectionsand destination information, which may be entered by a user of thecustomer appliance 300. The display screen 308 is used for displayingmenus, scanned images, and other information. In some embodiments thedisplay screen 308 is implemented as a color touch screen. In touchscreen embodiments the keypad 310 may be omitted. Accordingly the keypad310 is shown as an optional element. In order to provide audio output,e.g., to allow for the playback of recorded commentary and/or to supportaudio output as part of a video conference call, the appliance 300includes speaker 313. To support capture of audio, e.g., to allow forrecording of picture commentary and/or the input of voice as part of avideo conference call, the appliance 300 includes microphone (mic) 314.

Camera 316 is provided for capturing still images and/or video of theuser of the appliance 300. In the case of commentary provided relatingto a scanned photo, the camera can, and in some embodiments is, used tocapture an image or video of the person providing the commentary. Thecamera also supports video capture enabling video conference calls fromthe appliance 300. The camera 300 is usually mounted at a location onthe appliance housing from which the image of a user of the appliancecan be easily captured. In some embodiments, the camera 316 is mountedabove the display screen 308 as shown in FIG. 2.

Scanner 318 allows photos to be scanned. Scanned images may, and in someembodiments are, automatically reoriented prior to display thus allowingan image to be scanned in any direction with the appliance re-orientingthe image after scanning. In some embodiments scanner 318 is implementedas a small flatbed scanner capable of scanning 3×5 images. Such a sizeis well suited for standard photos. Larger scanning bed sizes may alsobe used. In other embodiments the scanner is implemented as a devicehaving a slot or feed input and the item to be scanned is moved over ascanning area. Accordingly, depending on the particular embodiment, thescanner 318 may be implemented in different formats. The scanner 318 canbe used a document scanner allowing documents to be scanned anddisplayed as part of a video phone conference.

The peripheral device input/output interface 320 serves to interface thedevice 300 with a variety of external optional peripheral devices aswell as a memory card slot 307. The memory card slot 307 allows formemory cards often used for the storage of photos to be read and/orwritten. Thus, not only can the appliance 300 be used to document andarchive physical photos which can be scanned, but can also be used toadd commentary to images which were captured by a modern digital camera.Thus, the appliance 300 remains relevant and useful even as a user maymigrate from film and photos to electronic cameras and electronicphotos.

Among the peripheral devices which are supported by the interface 320are various optional peripheral devices such as a printer, slide reader,VCR, 8 mm film reader, slide reader, etc. These peripheral devices maybe purchased by a user at the time of purchase of the appliance orlater, e.g., on an as needed basis. Peripheral devices added to thecustomer appliance 300 are automatically detected, configured ifnecessary and the added functionality and menu options made possible bythe addition of the peripheral device are automatically added by theappliance 300 to its set of menus. The peripheral device I/O interface320 may support USB devices. In addition to the interface 320, a sensorinterface 324 is provided for receiving local sensor input. The sensorinterface 324 may include a wired and/or wireless receiver/transmitter.A large variety of sensors may interface with the appliance via thelocal sensor I/O interface 324. Sensors which may be coupled to theappliance 300 via interface 324 include, e.g., health monitoringsensors, motion sensors, alarm sensors, etc. As discussed above,peripheral devices in the form of medical sensors may be paid for andcoupled to the appliance 300 at any time. Thus, a user of the appliance300 may purchase the appliance 300 for, e.g., photo and videoconferencing functions, and an insurance company may purchase andprovide the user a health monitoring device at some later time to beused with the appliance 300, e.g., at the insurer's expense. Healthmonitoring devices may include blood sugar level monitors, bloodpressure monitors, heart rate monitors, etc, which may be coupled to thedevice 300 via interface 324. Information provided by sensors viainterface 324 can, and in various embodiments are, uploaded by theappliance 300 to a network server for forwarding to a health careprovider, e.g., a doctor or health insurance provider. Thus, theappliance 300 can be used to support health monitoring functions inaddition to supporting video conferencing and photo achieving.

Appliance 300 can, and in some embodiments is, configured to detectvarious alarm conditions and take action in response to an alarmcondition. For example, in response to the failure to detect expectedmotion, or in response to detecting sudden motion indicative of a fall,the appliance 300 may initiate an audio or video conference with amonitoring service or healthcare provider which can then assess thesituation and make an informed decision as to whether or not to send,e.g., dispatch, medical help. Smoke and/or heat sensors may be used totrigger a fire alarm which, like a medical alarm, may trigger a videoconference call which can result in emergency service personal beingdispatched, e.g., fire fighters and/or police may be dispatched to thecustomer premise.

For cost reasons the memory 302 may be relatively small. The memory 302may be non-volatile and can be used to store various modules, e.g.,routines, which support various device functions. In addition memory mayhave the capacity to store a limited number of photos and correspondingaudio/video commentary which can then be uploaded to a network storagedevice via network interface 302.

In the FIG. 3 embodiment, memory includes tag and/or feature lists 325corresponding to scanned images which may be stored with the associatedtag/feature lists in memory portion 325 or in scanned image portion 327,a media archive module 326, a video conference call module 328, anonline collaboration module 330, remote/Internet upgrade module 332, aplug and play peripheral device support module 334, device controlroutines 336, communications routines 338 and stored data 340, e.g.,photos and/or audio/video commentary. The information stored in memoryportion 325 can be, and sometimes is feature and tag informationreceived from multiple different feature recognition systems with a setof information being stored for each scanned image which was subject tofeature recognition and tag list generation processing. Thus in someembodiments 325 includes a separate tag list and/or other informationfor each individual scanned image stored in memory portion 327.

While memory 302 may be somewhat limited in size for cost reasons, insome embodiments an optional hard disk 322 is included to provide forample storage for digital images, video and audio on the appliance 300.Cloud or network storage is also supported making optional hard disk 322less important in cases where reliable network connectivity isavailable.

Media archive module 326 controls the scanning, documentation (audioand/or video documentation) of images such as photos and physicaldocuments. As discussed above, photos can be scanned, stored in digitalform with captured audio and/or video commentary and distributed, e.g.,via the network interface 304 under control of media archive module 326.Video conference module 328 is responsible for handling video conferencecall establishment and video conference call maintenance. On-linecollaboration module 330 allows users to establish on-line collaborationsessions which may involve use of the video conference capabilitiesavailable from module 328 as well as document exchange capabilities madepossible by the availability of the scanner 318. Remote/Internet upgrademodule 332 allows for the appliance 300 to exchange data and/or controlinformation with a remote server via a communications network such asthe Internet or a public telephone network. Remote upgrade module 332makes it possible for the routines in memory 302 to be updated, added toor replaced via a remote network server. Thus, new applications androutines may be retrieved and installed automatically, e.g., as newperipheral devices are detected. Plug and play peripheral device supportmodule 334 is responsible for detecting new peripheral devices,retrieving corresponding applications or routines from the network ifrequired, automatically install the retrieved routines and for takingany other action required to automatically support various peripheraldevices attached to the customer appliance 300. The plug and playsupport made possible by module 334 allows a user to add supportedperipheral devices without have to be concerned with having to manuallyconfigure the appliance 300 to support the peripheral device.

Device control routines 336 include a variety of routines, includingalarm generation and detection routines, data storage control routines,etc. that support or control the device to operate in accordance withthe methods of the present invention.

Communications routines 338 support voice and data communications andenable communications sessions to be established via the appliance 300.Stored data 340 includes stored photos, voice and/or image datacorresponding to commentary relating to scanned or input photos ordocuments. The stored data may also include menu information and/orother system configuration information. The system configurationinformation may be preloaded and/or automatically updated as peripheraldevices as added and/or the device is reconfigured to support newapplications. Updating of configuration information stored in memory 302may be done automatically by a remote server coupled to the customerappliance 300 via a communications network. Data 340 may include alarmsettings which determine when a video conference call is to beinitiated, e.g., in response to a fall sensor, heat sensor, smoke alarmor another monitoring device which may supply signals to the customerappliance 300. Storage 302, e.g., memory, further includes an assemblyof components 354, e.g., assembly of software components, e.g., softwareroutines and/or software modules.

In view of the above discussion, it should be appreciated that theappliance of the present invention is easy to set up, simple to use,supports a wide variety of applications and can be updated remotelyand/or through the addition of add on peripheral devices which canincrease the number of supported functions. The appliance of the presentinvention supports enough functions that it can appeal to a wide rangeof family members and/or age groups. Since purchase of the appliance canbe justified by the many non-health related functions it supports,purchase of the appliance can be motivated without using the healthmonitoring features as a primary reason to purchase the appliance.Health care providers can cover the cost or supply health monitoringrelated peripheral devices and can take advantage of the remotereporting and alarm features supported by the appliance therebypotentially reducing the cost of health care services without saddlinghealth insurance companies with the cost of the communications interfaceand/or network related costs that might be associated with having toprovide a complete monitoring system.

FIG. 4 illustrates a system 400 implemented in accordance with thepresent invention in more detail than the FIG. 1 illustration. Asillustrated in FIG. 4, a variety of households 404, 474 include a homemanagement appliance 472 implemented in accordance with the presentinvention and one or more other electronic devices 470 which can be usedto access and distribute information, e.g., household member scheduleinformation, contact information, etc.

The households 404, 474 are coupled to the secure server 402 via securecommunications links such as the link 410. The secure links may passover a public communications network such as the network 414, e.g., theInternet or public telephone network, but are secured through the use ofone or more security techniques, e.g., encryption techniques. The link410 may be a VPN communication link. A firewall 412 protects the secureserver from security threats and also protects the home managementappliance 472 from security threats. Communications from the homemanagement appliance to Internet sites, communications services, E-mailservers, etc. are routed through the secure server 402 and firewall 412to protect the home management appliance 472 from security threatswithout imposing the need for the user of the home management appliance472 to manage or even be aware of the firewall 412.

The software and/or other applications are loaded onto the homemanagement appliance 472 via the secure server 402. In some embodimentsthe user is prohibited from loading applications or software onto thehome appliance 472 except via the secure server 402. Thus, the secureserver 402 can check applications before they are loaded onto the homeappliance 472 greatly reducing the threat of accidental loading ofviruses and also allowing the secure server 402 to make sure that onlycompatible applications are loaded onto a home appliance 472. The secureserver 402 may be responsible for updating appliance settings andconfiguration as applications are loaded and used in combination on thehome appliance 472. Since home appliance configuration and managementissues are implemented on the secure server 402 by the service provider,and the household member using the home management appliance 472 isshielded from many device management and configuration issues commonlyencountered when software and peripherals are loaded onto or added topersonal computers.

The secure server 402 includes various modules in addition to customerinformation 432, e.g., household information. The modules include abilling module 420, a service management module 422, a customer caremodule 424, an authentication module 426, a device management module428, a presence management module 430. The various modules can interactwith the home management appliance 472 via network interface 446 andalso with other external systems 408 via the interface 446. The customerinformation 424 includes information 436, 440 corresponding to each ofthe households 404, 474 serviced by the secure server 402. The householdinformation (household 1 information 436, . . . , household ninformation 440) includes information stored in a secure manner, e.g.,information stored in what is referred to as a secure vault (438, . . ., 442), respectively. The information in a households secure vault maybe encrypted and is normally backed up. A secure vault, e.g., securevault 442, used to store household information, e.g. for household n,may be distributed over a number of storage devices and may beimplemented using a secure cloud based storage embodiment and need notbe, but may be, implemented using storage internal to the secure server402.

External service provider systems 408 which may be accessed from thehome management appliance 472 via the secure server 402 include acollaboration system 450, one or more telephone service provider systems456, 458, one or more instant message service provider systems 452, 454,an application store 463, and one or more internet service providersystems 460, 462. Various other external devices such as ad server 464,a Google server 466 and a search engine server 468 may also be accessedvia the home management appliance 472 and secure server 402. While notshown in FIG. 4, one or more health care service provider systems 114may also be accessed via home management appliance 472 and secure server402.

A secure vault, e.g. secure vault 438, can be used to store medicalrecords, receipts, business records, photos, schedule information, to dolists, etc. with the amount of access to such information beingcontrolled and/or restricted based on information entered via the homemanagement appliance 472 of the household, e.g. household 1 404, towhich the secure vault, e.g., secure vault 438, corresponds. A user ofthe home management appliance 472 can manage a households communicationsservices, including the porting of telephone numbers and change oftelephone service providers, directly from the home management appliance472 by providing change/service management information to the secureserver 402. The service provider operating the secure server 402 thencontacts the various service providers which need to make changes toimplement the requested communications service changes in a securemanner eliminating the need for the user of the home managementappliance 472 to directly contact the individual communications serviceproviders affected by the requested change in communications service.E-mail, instant messaging and other communications services used byhousehold members can also be managed in a secure manner form homemanagement appliance 472.

FIG. 5 illustrates a home appliance 500 which may be used as the homeappliances shown in FIGS. 1 and 4. The home appliance 500 includes adisplay 502, base unit 506 and scanner 510. The display device 502 mayinclude a camera 504 and/or microphone (mic) 505 and may be touchsensitive. The scanner 510 may be integrated with the base unit 506 ordetachable. The base unit 506 includes interfaces 512, 514 for couplingthe base unit 506 to external peripheral devices such as healthmonitoring apparatus 520 and external media 522. The base unit 506further includes a memory card slot 509. The base unit 506 also includesan interface 511 for securely coupling the base unit 506 to the secureserver 102 via a communications network 516. Interface 511 includes areceiver 524 and a transmitter 526.

FIG. 6 illustrates an exemplary home appliance 600 which may be used asthe home appliances shown in FIGS. 1 and 4. Exemplary home appliance 600includes a display 502, a base unit 506 and scanner 510. FIG. 6 showshow, in one embodiment the display 502, base unit 506 and scanner 510may be detached from one another. The various components (502, 506, 510)may communicate wirelessly with one another, e.g., via wirelessinterfaces (602, 604, 606), and corresponding antennas (603, 605, 607),respectively. Wireless interface 602 includes a wireless receiver 610and a wireless transmitter 612. Wireless interface 604 includes awireless receiver 614 and a wireless transmitter 616. Wireless interface606 includes a wireless receiver 618 and a wireless transmitter 620. Thedisplay device 502 may be a tablet type device including a touchsensitive screen, display capability, some processing capability and theability to wirelessly interact with the base unit 506, e.g. a basestation, and via the base station 506, including wireless interface 511′and antenna 513′ and wireless communications link 516′, the secureserver 102. Wireless interface 511′ includes a wireless receiver 622 anda wireless transmitter 624.

FIG. 7 illustrates a household control panel 700 implemented inaccordance with one embodiment of the present invention. The householdcontrol panel 700 may be displayed on the display 502 and accessed viathe home management appliance. Functions and/or services provided viathe control panel may be implemented partially or fully on the secureserver 102. Exemplary household control panel 700 includes a settingportion 704, a tools portion 706 and a command mode services portion708. The settings portion 704 includes administrator settings 712, andhousehold settings/preferences corresponding to each household member(household member 1 setting/preferences 712, . . . , household member nsettings/preferences 714).

FIG. 8 illustrates a family command center 800 which may be displayed onand accessed via the home management appliance. The family commandcenter 800 is an application and corresponding display interface whichallows for the management of a wide variety of household informationincluding contacts 812, reminders 811, a calendar 808, scanned receipts810, call information 804, tools 816 and a bulletin board 806 which canbe used for posting notes and/or other information intended for multiplefamily members.

FIG. 9 is a drawing illustrating a secure server 900 which may be usedas the service provider secure server (102 or 402) of the systems shownin FIGS. 1 and 4. Secure server 900 includes a wired and/or wirelessinterface 902, I/O device 904, processor 906, e.g., a CPU, assembly ofcomponents 908, e.g., an assembly of hardware components, e.g. circuits,network interface 446, an memory 912 coupled together via a bus 910 overwhich the various elements may interchange data and information. Memory912 includes billing module 420, service management module 422, customercare module 424, authentication module 426, device management module428, presence management system 430, customer information 432, andassembly of components 913, e.g., an assembly of software components,e.g. software routines and/or software modules. Customer information 432including sets of household information corresponding to a plurality ofdifferent households (household 1 data/information 436, . . . ,household n data/information 440. Household 1 data/information 436includes secure vault 438. Household n data/information 440 includessecure vault 442.

Wired and/or wireless interface 902 includes one or more or all of:wired receiver (RS) 914, wired transmitter (TX) 916, wireless receiver(RX) 918 coupled to receive antenna 922, and wireless transmitter (TX)920 coupled to transmitter 924. In some embodiments, the wirelessreceiver 918 and wireless transmitter 920 use the same antenna or sameset of antennas. Network interface 446 includes a receiver (RX) 926 anda transmitter (TX) 928.

FIG. 10 illustrates how asset ownership and/or access rights may be andsometimes are automatically changed in response to one or more lifeevents that can be specified by a user and/or determined or verifiedbased on one or more scanned documents.

The diagram 1000 shows a variety of asset ownership/access states 1120,1106, 1110, 1102, 1114 and the corresponding life events that can beindicated by a user and/or automatically detected by scanning a documentsuch as a marriage certificate, death certificate, birth certificate,divorce document, lease indicating a child has moved to a new address.Arrows are used to indicate transitions, e.g., changes in ownershipand/or access that can be and sometimes is implemented automatically inresponse to detection of the corresponding event.

For example state 1120 represents the existence of a set of digitalassets which are owned by a person represented by the letter B.Ownership of the stored assets, e.g., records, pictures, legaldocuments, audio records, etc. transitions in step 1121 to a joint assetownership state in response to detection of a marriage betweenindividual A and individual B. State 1106 indicates the state of jointlyowning assets stored in the digital vault maintained for the familyincluding married individuals A and B. The ownership state of the assetsmaintained for the coupled A, B changes in response to detecting one ormore conditions such as death of individual A, death of individual B ordivorce of A& B.

In the case of death of individual A ownership of the assets stored inthe digital store that are jointly owned by A and B automaticallytransitions in step 1122 to ownership by individual B 1120. In the caseof death by individual B ownership of assets stored in the digital storethat are jointly owned by A and B automatically transitions in step 1124to ownership by individual A 1102.

In the case of divorce of married couple assets which are jointly owned1106 are divided between A & B based on previous ownership and/or adivorce document. Steps 1126 and 1127 represent the transition of assetsform the joint ownership state 1106 to the individual ownership states1120, 1102 as a result of divorce.

Detection of a birth or adoption of a child can, and sometimes does,trigger automatic creation of a secure data store in the family digitalvault for the child while also automatically granting the child accessrights to some or all of the parents data stored in the family datavault, e.g., maintained by the secure server 402. These operations arerepresented by step 1108.

A child leaving a home, e.g., as indicated by the scanning of a leasewith the child's name and new home address on it, can, and sometimesdoes, trigger an automatic change in the child's right to access contentcorresponding to his/her parents as represented by step 1112. After adetermination is made that a child has left the family home the child isstill permitted to access some portions of the family vault, e.g., theportions including general family history, family photos and the child'sown personal information but may be, and sometimes is, restricted fromaccess to some other content in the family vault the child had access toprior to leaving the home. The content which the children are to haveaccess to can be, and sometimes are, determined by general rulesspecified by one or both the parents which can be applied uniformly toone or more children whenever a child is detected on indicated to haveleft the home.

The ownership and access changes may be, and sometimes are, determinedby the customer home appliance 300, e.g., under control of the processorincluded therein, based on user input and/or one or more scanneddocuments and then communicated via a communications network to thesecure server which maintains the secure family vault. In otherembodiments the home appliance or another device provides user inputand/or scanned documents to the secure server 402 which then makes assetownership and/or asset access changes, e.g., in accordance with thediagram of FIG. 10 and/or the method shown in FIG. 11.

FIG. 11 shows how FIGS. 11A, 11B and 11C can be combined to form a flowchart showing steps relating to asset storage, ownership and/or accesscontrol which are used in various embodiments of the present inventionto manage contents of a digital data store that may be, and sometimesis, a remote cloud based data store including family information.

While document scanning is normally done in a home using a scannerincluded in or attached to a device in the home, the access control andownership decisions can be made by the processor of the home device orby the processor of the secure server depending on the particularembodiment.

Thus depending on the particular embodiment one or more of the stepsshown in FIG. 11 can be implemented by the home management appliance 472or another home device and/or by components in the secure sever 402which controls the secure server to operate in accordance with theinvention.

FIG. 11, comprising the combination of FIG. 11A, FIG. 11B and FIG. 11C,is a flowchart 1200 of an exemplary method in accordance with anexemplary embodiment. Operation starts in step 1200 in which anexemplary system, e.g., a system including a secure server, a pluralityof household management appliances, and one or more external systemprovider systems, is powered on and initialized. Operation proceeds fromstart step 1202 to step 1204.

In step 1204 digital assets, e.g. documents, images and/or information,corresponding to a group of associated individuals, are stored in astorage device. Operation proceeds from step 1204 to step 1206.

In step 1206 rights information, e.g., information indicating ownership,access rights, and/or use privileges with regard to stored digitalassets, e.g., on a per individual basis with different family membershaving different rights, are stored. Operation proceeds from step 1206to step 1208.

In step 1208 information indicating rights changes to be made based upondetection of one or more events associated with a family, e.g.,marriage, divorce, death of parent, birth of child, death of child,e.g., are stored. Step 1208 includes steps 1210 and 1212. In step 1210predetermined rights changes for one or more life events are stored. Instep 1212 user, e.g. family member, specified rights changes, e.g.inheritance of ownership of assets or access rights upon death,marriage, etc., are stored. Operation proceeds from step 1208 to step1210.

In step 1214 monitoring for input is performed. Operation proceeds fromstep 1214 to step 1216. In step 1216, if user or other input wasreceived, then operation proceeds from step 1216 to step 1218. However,if no user or other input was received, then operation proceeds fromstep 1216 to the input of step 1214 for additional monitoring.

Returning to step 1218, in step 1218 if user input was received, thenoperation proceeds from step 1218 to step 1220; otherwise, operationproceeds from step 1218, via connecting node B 1219 to step 1240.

Returning to step 1220, in step 1220 if the user requested change isexplicit in rights information, then operation proceeds from step 1220,via connecting node A 1223 to step 1228; however, if the user requestedchange is not explicit in rights information, the n operation proceedsfrom step 1220 to step 1222.

In step 1222 a check is made as to whether or not the user has theauthority to make the change. If the check of step 1220 determines thatthe user has the authority to make the requested change, then operationproceeds from step 1222 to step 1224, in which the requested change tothe rights information is made. However, if the check of step 1220determines that the user does not have the authority to make therequested change, then operation proceeds from step 1222 to step 1226,in which the request change is not made. Operation proceeds from step1226 to the input of step 1214, in which additional monitoring for inputis performed.

Returning to step 1228, a determination is made as to whether or not thereceived user input is indicating a life event. In step 1228 if thereceived user input is indicating a life event, then operation proceedsfrom step 1228 to step 1230; otherwise, operation proceeds from step1228 to step 1232.

Returning to step 1230, ins step 1230, a determination is made as towhether or not life event verification is required. If life eventverification is required, then operation proceeds from step 1230 to step1234; otherwise operation proceeds from step 1230 to step 1238.

Returning to step 1234, in step 1234 a scanned document, e.g., amarriage certificate, birth certificate, death certificate, etc., isautomatically checked for information verifying life event. Operationproceeds from step 1234 to step 1235 in which the life event isoptionally verified by accessing a government or other informationdatabase. Operation proceeds from step 1235 to step 1236.

In step 1235 a determination is made as to whether or not the checkconfirmed that the life event occurred. If the determination is that thecheck confirmed the life event, then operation proceeds from step 1236to step 1238; otherwise, operation proceeds from step 1236, viaconnecting node C 1233, to the input of step 1214 for additionalmonitoring.

Returning to step 1238, in step 1238 right changes or asset ownershipchanges corresponding to the life event indicated by the user areimplemented.

Returning to step 1234, in step 1232 user input is responded to, e.g.,access is provided to requested content if the user is authorized toaccess the requested content. Operation proceeds from step 1232, viaconnecting node C 1233, to the input of step 1214, for additionalmonitoring.

Returning to step 1240, in step 1240, it is determined if the otherinput which was received is document input. If the received input isdocument input, then operation proceeds from step 1240 to step 1242;otherwise operation proceeds from step 1240 to step 1243, in whichreceived input is processed. Operation proceeds from step 1243. viaconnecting node C 1233 to the input of step 1214 for additionalmonitoring.

Returning to step 1242, in step 1242 the document contents areautomatically reviewed. Operation proceeds from step 1242 to step 1243.

In step 1243 the document is processed, e.g., the document type, e.g.photo, document invoice, etc., is detected, official certificateidentity information, and/or objects are detected and stored inaccordance with detected information and/or based on document type.Operation proceeds from step 1244 to step 1246.

In step 1246 it is determined if the document indicates or confirmsoccurrence of a lifer event, e.g. marriage, divorce, death, birth, etc.If the determination is that the document does not indicate or confirmoccurrence of a life event, then operation proceeds from step 1246, viaconnecting node C 1233, to the input of step 1214 for additionalmonitoring. However, if the determination is that the document doesindicate or confirm occurrence of a life event, then operation proceedsfrom step 1246, to optional step 1248 or to step 1250. In step 1248, adetermination is made as to whether or mot the document is authentic. Ifthe determination is that the document is not authentic, then, operationproceeds from step 1248, via connecting node C 1233 to the input of step1214 for additional monitoring. However, if the determination is thatthe document is authentic, then, operation proceeds from step 1248, viaconnecting node C 1233 to step 1250. In step 1250 rights changescorresponding to the life event indicated in the scanned document areimplemented. Operation proceeds from step 1250, via connecting node C1233, to the input of step 1214 for additional monitoring.

FIG. 12 shows the steps of a method of processing a physical image inthe form of a photo or document to digitize the image and associate tagswith the image for storage and to facilitate possible future word basedimage searching and image retrieval. The method shown in FIG. 12 may be,and sometimes is, implemented by an appliance such as one of theappliances 108, 111, 200, 300 shown in FIGS. 1, 2 and 3 which caninteract with the artificial intelligence systems 97, 98 and/or 99 toprovide them scanned images and receive there from reports relating tofeatures recognized in the scanned images. Individual reports receivedfrom the AI systems can, and sometimes do, provide a list of featuresrecognized by the system to be in the image and, for each detectedfeature a list of one or more tags and a confidence level indicating theprobability that the feature is in the scanned image which wasprocessed. As will be discussed below, this information can be, andsometimes is, used to determine which tags are associated with an imageas part of an automated process that may potentially involve some userinput in terms of an operator viewing the scanned document confirmingthat one or more features are present and/or confirming a tag or synonymof a tag that should be associated with a scanned image that is beingstored. Tags may be, and sometimes are, individual words but can also bephrases.

The method 1251 shown in FIG. 12 starts in start step 1252 with a homeappliance being powered on and the processor beginning to implement aroutine which controls the home appliance 108, 111, 300 or 500 toimplement the method shown in FIG. 12. Operation proceeds from step 1252to scan step 1253 where an image, e.g., a document or photograph, isscanned, e.g., using scanner 318 which may be integrated into the homeappliance or attached to the home appliance.

Operation proceeds from scan step 1253 to step 1254 wherein theprocessor 306 and/or memory 302 of the home appliance, e.g., exemplaryappliance 300 receive the scanned image and then communicate it via theinterface 304 and/or communications network 104 to the AI systems 97,98, 99.

The AI systems 97, 98, 99 receive and process the scanned image in steps1256, 1258, 1260, respectively. The first AI system 97 returns a firstset of information 1262 including a list of identified features, a tagor tags for each individual identified feature and a correspondingconfidence level that the individual feature is present in the scannedimage. The second AI system 98 returns a second set of information 1264including a list of identified features, a tag or tags for eachindividual identified feature and a corresponding confidence level thatthe individual feature is present in the scanned image. The third AIsystem 99 returns a third set of information 1266 including a list ofidentified features, a tag or tags for each individual identifiedfeature and a corresponding confidence level that the individual featureis present in the scanned image. Since the different AI systems rundifferent feature recognition algorithms the results provided by thethree systems may be different with some of the systems recognizing oneor more features which were not recognized by one of the other AIsystems 97, 98 or 99.

The results 1262, 1264, 1266 of scanned image analysis are returned fromthe AI systems 97, 98, 99 to the appliance 108. Steps 1268′, 1268″,1268′″ represent the home appliance 108 receiving the feature detectionresults from the AI systems 97, 98, 99 respectively. In some embodimentsthe results from each AI system are provided in list form for aprocessed image. The list returned by the AI system includes a list offeatures which were identified and for each identified feature adetection probability also sometimes referred to as a confidence measureor value and one or more tags corresponding to the identified feature.For example if AI system 1 97 identifies a dog as a first detectedfeature in a first image that was received form the home appliance itwould include in a list that a dog feature was detected, a probabilitythat the dog feature was accurately detected, e.g., a value 0.9indicating a 90% certainty that the dog feature was present, and one ormore tags corresponding to the dog feature such as the word “dog” and/orother terms such as “German Sheppard” or other names for one or morebreads of dogs. The list returned by the AI system 97 and received instep 1268′ may include other features, a corresponding confidence valuefor each individual feature and one or more tags for each identifiedfeature.

In step 1269 the home appliance identifies consensus features, e.g., oneor more features which were individual identified by more than one ofthe various AI systems 97, 98, 99, e.g., features that the AI systemsagree are present in the scanned image. Consensus of the AI systemsmaybe and sometimes is evidenced by multiple AI systems identifying thesame feature, e.g., a “dog feature”. In step 1269 the features in theresults 1262, 1264 and 1266 are compared and features which wereidentified by each of the AI systems are identified as consensusfeatures when multiple AI systems agree that these features are present.For each identified consensus feature the corresponding confidence levelprovided by each AI system and tags are supplied to step 1270. In step1270 an overall confidence level is determined, e.g., by the homeappliance 108, for each individual consensus feature based on theprobabilities returned by each of the multiple AI systems 97, 98, 99that detected the feature in the scanned image being processed.

Step 1270 may be, and sometimes is, implemented by the processor of theappliance making a call to a overall confidence level determinationsubroutine such as the one shown in FIG. 13 to which the consensusfeature information is supplied for processing.

In other embodiments the overall confidence level for a particularconsensus feature may be and sometimes is generated by forming a simpleaverage or a weighted average of the probability, e.g., confidence,values returned by the systems which detected the particular feature.

The method 1300 of determining an overall confidence level for aconsensus feature will now be discussed with reference to FIG. 13. Themethod 1300 starts in step 1302 when the routine is called, e.g., by themethod 1251 shown in FIG. 12. Each consensus feature is processedindependently with the process proceeding from step 1302 to step 1304for each identified consensus feature. Thus the processing shown insteps 1304 to 1312 will be performed for each consensus feature.

In step 1304 the average feature detection rates for each of the AIsystems AI 1 97, AI 2 98 and AI 3 99 which detected the consensusfeature and returned results are taken into consideration and averagedto generate an average detection rate that serves as an prior input tostep 1306 as part of a Bayesian inference process. The average featuredetection rates are, in some embodiments, rates based on past testingand not based on the processing of the scanned image. Such rates areconsidered prior knowledge that can be used as an input to a Bayesianprocess, where the probability is updated based on the confidence levelreported by one or more AI systems that the consensus feature to whichthe processing in steps 1304 to 1310 relates was actually detected.

In step 1306 a first posterior density probability is generated from theaverage probability supplied by step 1304 and the confidence 1305indicated by the first AI system 97 that the feature to which theprocessing relates was detected by the first AI system 97 in the scannedimage.

The probability generated in step 1306 is supplied as a prior to step1308 which also receives the probability that the consensus feature wasdetected by the second AI system 98. In step 1308 a second posteriordensity probability is generated from the posterior density probabilitysupplied by step 1306 and the confidence 1307 indicated by the second AIsystem 98 that the feature to which the processing relates was detectedby the second AI system 98 in the scanned image.

The probability generated in step 1308 is then supplied as a prior tostep 1310 which also receives the probability that the consensus featurewas detected by AI system 99. In step 1310 a third posterior densityprobability is generated from the posterior density probability suppliedby step 1308 and the confidence 1309 indicated by the third AI system 99that the feature to which the processing relates was detected by thethird AI system 99 in the scanned image.

The probability generated in step 1310 takes into consideration theprobabilities indicated by each of the 3 AI systems (97, 98, 99) thatthe feature to which the processing relates is in the scanned image andis considered an overall confidence level for the consensus feature. Theoverall confidence level for the consensus feature to which theprocessing relates is returned to the processor which called thesubroutine 1300 in steps 1312 and can be used, as discussed below tomake a decision as to which tag or tags should be associated with animage.

Since steps 1304 to step 1312 are performed for each consensus featurethat was detected, at the end of the processing shown in FIG. 13, theapparatus implementing the method will have an overall confidence levelfor each consensus feature being processed and operation will return tothe point from which method 1300 was called.

Referring once again to FIG. 12, at the end of step 1270, the apparatus,e.g., home appliance 108, will have an overall confidence level, e.g.,probability, for each individual consensus feature that was identifiedin step 1269. Operation proceeds from step 1270 to step 1272 in whichone or more tags are determined to include in a tag list to beassociated with the scanned image. The tag list, in some embodiments,includes at least one tag for each consensus feature having a high levelof certainty of being present in the scanned image as evidenced byeither: i) having an overall confidence level over a threshold value,e.g., a value which may be a preset value and/or ii) confirmed by asystem operator to be present in the scanned image.

In some embodiments whether to include a tag in a tag list correspondingto an image is implemented based on a simple threshold basis with thetag being included in the tag list being created when the overallconfidence value corresponding to the feature with which the tag isassociated exceeds a threshold level, e.g., a predetermined thresholdlevel. In cases where the overall confidence value exceeds the thresholdlevel the tag and corresponding feature are included in the tag listthat is associated with the scanned image. However if the overallconfidence value associated with a feature and tag or tags is at orbelow the threshold, in some embodiments the feature and/orcorresponding tags are not included in the tag list for the scannedimage.

More complicated approaches to deciding whether to include a tag in atag list can be used with, in some embodiments limited user input beingused to facilitate the tag association processes. FIG. 14 shows oneapproach to implementing step 1272 which can involve some user inputdepending on the embodiment and/or overall confidence level associatedwith a tag and corresponding identified consensus feature. Accordingly,to facilitate an understanding of invention, FIG. 14 will now bediscussed.

The method 1400 shown in FIG. 14, starts in step 1402 in response to acall, e.g., from routine 1251, and with the information 1401 includingthe overall confidence level information for features and thecorresponding tags being supplied to the routine 1400. Steps 1404 to1418 are performed for each individual consensus feature to determine ifthe tag or tags corresponding to the feature should be associated withthe scanned image.

In step 1404 a determination is made as to whether the overallconfidence level corresponding to the feature is below a minimumconfidence level threshold, e.g., a threshold indicating that theprobability of the feature being present is sufficiently low that thefeature should not be associated with the image and thus the tagscorresponding to the feature should not be associated with the image. Instep 1404 if the overall confidence level of the feature is below theminimum confidence level threshold operation proceeds from step 1404 todiscard tag/feature step 1413. In step 1413 the tag or tagscorresponding to the feature are discarded and the feature and tags willnot be associated with the scanned image, e.g., due to the low level ofconfidence that the feature is in the scanned image. Operation proceedsfrom step 1413 to step 1420 where processing regarding the individualfeature is halted. While the processing of the individual feature insubroutine 1400 stops in step 1420 processing return to the main routineand continue in routine 1251 which called subroutine 1400.

In step 1404 if the decision was that the overall confidence level ofthe feature was not below the minimum confidence level thresholdoperation will proceed to step 1406. In step 1406 the overall confidencelevel is compared to an upper confidence level threshold to determine ifit is over the upper confidence level threshold indicating that thefeature is highly likely to be in the scanned image. If in step 1406 theoverall confidence level of the feature is determined to be over theupper confidence level threshold operation proceeds to step 1414 inwhich a decision is made to include a tag or tags corresponding to thefeature in information being associated with the scanned image asindicated by a decision to designate the tag or tags associated with thefeature as confirmed tags, e.g., tags which correspond to a featureconfirmed to be in the scanned image and thus which should be associatedwith the image.

In step 1406 if it was determined that the overall confidence level ofthe feature did not exceed the upper confidence level threshold,operation proceeds from step 1406 to step 1408 in which an attempt ismade to obtain input from a user as to whether or not the feature ispresent in the scanned image. In step 1408 the tag, e.g., word,identifying the feature, is presented to an operator of the appliance asa suggested tag for the scanned image and the operator, is requested toconfirm that the tag should be used with the scanned image. The scannedimage may be, and sometimes is, presented to the operator with the tagon a display, e.g. display 204 or display 308, at the time operator ofthe apparatus is requested to confirm that the tag should be associatedwith the scanned image.

In step 1410 a user response is received indicating whether thesuggested tag should be used, e.g., associated with the scanned image,or not used. If the operator, e.g., person who scanned the image,indicates that the tag should not be used, operation proceeds from step1412 to discard tag/feature step 1412 and the tag and feature will notbe associated with the scanned image. Since the tag corresponds to thefeature confirmation of a tag by an operator also servers as humanconfirmation that the corresponding feature to which the tag relates isin the scanned image.

However, if in step 1412 the user confirms that the tag should be used,e.g., associated with the scanned image because the feature indicated bythe tag is in the scanned image, operation proceeds from step 1412 tostep 1414.

In step 1414 the tag that was confirmed by the operator is designated asa confirmed tag and operation proceeds from step 1414 to step 1416. Instep 1416 a synonym or synonyms corresponding to the confirmed tag areidentified, e.g., via a look up of synonyms in a synonym listing storedin memory. While synonyms may be available for some words they may notbe available for all words. Accordingly, in step 1416 synonyms will beidentified when they exist which will be in the case of many tags.Operation proceeds from step 1416 to step 1418 where the tag or tags,e.g., words, identifying a confirmed feature are added to a confirmedtag list corresponding to the scanned image. In this way the confirmedtag corresponding to a detected feature is associated in memory with thescanned image. Operation proceeds from step 1418 to step 1420 whichrepresents the stopping point of processing performed in method 1400corresponding to an individual consensus feature. While in someembodiments synonyms are added automatically for confirmed tags, inother embodiments the operator is requested to review the identifiedsynonyms and confirm they should be associated with the scanned imageprior to the individual synonyms being associated with the scannedimage.

As tags corresponding to different consensus features are processed inthe method 1400, tags corresponding to different consensus features areadded to the confirmed tag list associated with a scanned image,multiple detected features are automatically associated with the scannedimage in a reliable manner.

With the completion of the processing of consensus features in FIG. 14and the tags to be associated with the scanned image having beendetermined, operation returns to step 1272 shown in FIG. 12 and thenproceeds to step 1274 in which the confirmed tag list associated withthe scanned image is stored in memory, e.g. with the scanned image. Thetag list may include simply the confirmed tags associated with thescanned images but may and sometimes also does include one or moreconfirmed features to which one or more listed tags correspond. The taglist and scanned image may be, and in some embodiments is, communicatedin step 1276 by the apparatus 300, e.g., home appliance 108, to thesecure server 102 for storage. The tag list and scanned image may be andsometimes is also main tined in the memory of the apparatus 300. Thusdepending on the embodiment the secure server and/or home appliance maystore scanned images and corresponding tag lists that can be used torespond to user word queries seeking images based on the input of a wordor words which may be stored in a tag list corresponding to a storedimage.

With the tag lists and corresponding scanned images having been storedin the secure server (102) and home appliance (300 or 300), a user canenter one or more words in a query and obtain, as part of a queryresult, one or more corresponding stored images. In at least someembodiments an image is determined to correspond to a query if a wordentered as part of the query is included in the tag list associated witha stored image. For example, a query including the work “dog” wouldreturn a query result which includes the first image which was found toinclude a dog image as a feature and for which the tag “dog” was storedin memory in the tag list corresponding to the image including the dogfeature.

Step 1278 relates to the use of the stored set of scanned images andassociated tags to respond to a user query which in some embodiments isentered using the keyboard of the home appliance 300 or 108. In step1278 a search query is received. As noted above the search query maybeand sometimes is a word. The following receipt of the search query instep 1280 a search is conducted using the word received in the query asa search term. The search is conducted by the home appliances eitherusing the stored tag lists included in its memory or by communicatingthe word to be use in the search to the secure server which includes thetag lists and images. Thus, depending on the embodiment the searchperformed in 1280 can be conducted at the home appliance 300, 108 or atthe secure server. As part of conducting the search, the word in thequery is compared to tags in the stored tag lists and stored scannedimages having a tag matching the word being searched are returned inresponse to the query. In step 1282 the home appliance and/or secureserver which implemented the search responds by providing the user thescanned image or images having a tag corresponding to the searched wordor words. The home appliance displays in step 1282 the image or imagesreturned as search results thereby providing the user of the appliance300 or 108 a response to the query.

Having responded to a query operation is shown as stopping in step 1284but it should be appreciated that the steps of FIG. 12 can be used toprocess additional images and/or quires on an as needed basis inresponse to a user scanning images using the home appliance 300 or 108or entering an image search query into the home appliances 300 or 108.

In step 1278 a device, e.g., the secure server 102 or appliance 300,receiving a search query response to the search query including a wordmatching a tag associated with the scanned image by providing thescanned image, e.g., to a display or device which is used to present thescanned image to the person who entered the query. As should beappreciated when the person who entered the query is the operator of theappliance, the appliance or server 102 will respond to the query byproviding the scanned image having an associated tag matching the searchterm and causing the image to be displayed on a display to the operatorwho entered the query.

The method stops in steps 1280 with the system, appliance or devicesbeing used to implement the method 1251 being powered down.

In view of the above, it should be appreciated that the appliance of thepresent invention offers numerous advantages over personal computerbased systems and/or dedicated health care or photo scanning systems.

FIG. 15 illustrate an exemplary artificial intelligence (AI) imagefeature identifier device 1500, e.g. server, in accordance with anexemplary embodiment. Exemplary AI image feature identifier device 1500is, e.g., any of the AI image feature identifiers (AI-1 image featureidentifier 97, AI-2 image feature identifier 98, AI-3 image featureidentifier 99) of system 100 of FIG. 1.

Exemplary AI image feature identifier 1500 includes I/O device(s) 1502,e.g., a keyboard and a display, a network interface 1504, a wirelessinterface 1505, a processor 1506, e.g., a CPU, an assembly of components1508, e.g., an assembly of hardware components, e.g. circuits, an memory1510 coupled together via a bus 1512 over which the various elements mayinterchange data and information. Memory 1510 includes an assembly ofcomponents 1514, e.g., an assembly of software components, e.g.,software routines, software subroutines, software modules and/or apps,and data/information 1516.

Network interface 1504 may, and sometimes does, couples the AI imagefeature device 1500 to a secure server, the Internet, communicationsnetworks, and/or other network nodes, e.g., a household managementdevice at a customer premises. Wireless interface 1505 includes awireless receiver 1507 coupled to receive antenna 1511 and a wirelesstransmitter 1509 coupled to transmit antenna 1513. Wireless interface1505 may, and sometimes does, couples the AI image feature device 1500to a secure server, the Internet, communications networks, and/or othernetwork nodes, e.g., a household management device at a customerpremises.

FIG. 16, comprising the combination of FIG. 16A and FIG. 16B, is adrawing of an exemplary assembly of components 1600, comprising thecombination of Part A 1601 and Part B 1603, which may be included in adevice or combination of devices, e.g. secure server, a householdappliance, and/or one or more artificial intelligence image featureidentifier device(s), in accordance with an exemplary embodiment.

FIG. 16 is a drawing of an assembly of components 1600, which may beincluded in a device(s), e.g., an appliance in a home and/or in a secureserver and/or in artificial intelligence image feature identifierdevice(s), in accordance with an exemplary embodiment.

In some embodiments, a device including one or more or all of thecomponents in assembly of components 1600 is an appliance in a home, forexample, the appliance is one of appliance 108 or 111 or FIG. 1,appliance 200 of FIG. 2, appliance 300 of FIG. 3, appliance 472 of FIG.4, appliance 500 of FIG. 5, and/or appliance 600 of FIG. 6. Inembodiments, the device includes or is coupled to one or more AI featureidentifier(s).

In some embodiments, a device including on or more or all of thecomponents in assembly of components 1600 is a secure server, e.g.secure server 102, secure server 402 or secure server 900, whichreceives user and document input from a home management appliance or oneor more other devices, e.g., device 470, which include or are coupled toa scanner, e.g. scanner 510. In some embodiments, the secure server 102includes or is coupled to one or more AI feature identifier(s).

In some embodiments, a device including some of the components inassembly of components 1600 is an artificial intelligence (AI) featureidentifier device, e.g. any of AI features Identifiers 97, 98, 99 ofFIG. 1 or AI feature identifier device 1500 of FIG. 15.

The components in the assembly of components 1600 can, and in someembodiments are, implemented fully in hardware within a processor, e.g.,processor 306 or 906 or 1506, e.g., as individual circuits. Thecomponents in the assembly of components 1600 can, and in someembodiments are, implemented fully in hardware within the assembly ofhardware components 352 or 908 or 1508, e.g., as individual circuitscorresponding to the different components. In other embodiments some ofthe components are implemented, e.g., as circuits, within processor 306or 906 or 1506 with other components being implemented, e.g., ascircuits within assembly of components 352 or 908 or 1508, external toand coupled to the processor 306 or 906 or 1506. As should beappreciated the level of integration of components on the processorand/or with some components being external to the processor may be oneof design choice. Alternatively, rather than being implemented ascircuits, all or some of the components may be implemented in softwareand stored in the memory 302 or 912 or 1510 of the device 300 or 900 or1500, with the components controlling operation of device 300 or 900 or1500 to implement the functions corresponding to the components when thecomponents are executed by a processor e.g., processor 306 or 906 or1506. In some such embodiments, the assembly of components 1600 isincluded in the memory 302 or 912 or 1510 as part of assembly ofsoftware components 354 or 913 or 1514. In still other embodiments,various components in assembly of components 1600 are implemented as acombination of hardware and software, e.g., with another circuitexternal to the processor providing input to the processor which thenunder software control operates to perform a portion of a component'sfunction.

When implemented in software the components include code, which whenexecuted by a processor, e.g., processor 306 or 906 or 1506, configurethe processor to implement the function corresponding to the component.In embodiments where the assembly of components 1600 is stored in thememory 302 or 912 or 1510, the memory 302 or 912 or 1510 is a computerprogram product comprising a computer readable medium comprising code,e.g., individual code for each component, for causing at least onecomputer, e.g., processor 306 or 906 or 1506, to implement the functionsto which the components correspond.

Completely hardware based or completely software based components may beused. However, it should be appreciated that any combination of softwareand hardware, e.g., circuit implemented components may be used toimplement the functions. As should be appreciated, the componentsillustrated in FIG. 16 control and/or configure the device 300 or 900 or1500 or elements therein such as the processor 306 or 906 or 1506, toperform the functions of corresponding steps illustrated and/ordescribed in the method of one or more of the flowcharts, signalingdiagrams and/or described with respect to any of the Figures. Thus theassembly of components 1600 includes various components that performfunctions of corresponding one or more described and/or illustratedsteps of an exemplary method, e.g., the method of flowchart 1251 of FIG.12, the flowchart 1300 of FIG. 13 and the flowchart 1400 of FIG. 14.

Assembly of components 1600 includes a component 1653 configured tocontrol a scanner to scan an image, a component 1654 configured tooperate a device to receive a scanned image, a component 1655 configuredto operate a device to communicate a scanned image to AI systems forprocessing, a component 1656 configured to process an image using a 1st.AI recognition system to generate a 1st set of identified features,corresponding feature tags and corresponding feature probabilities, acomponent 1658 configured to process an image using a 2nd AI recognitionsystem to generate a 2nd set of identified features, correspondingfeature tags and corresponding feature probabilities, and a component1660 configured to process an image using a 3nd AI recognition system togenerate a 3rd set of identified features, corresponding feature tagsand corresponding feature probabilities.

Assembly of components 1600 further includes a component 1668 configuredto receive a set of identified features (e.g., 1st set of identifiedfeatures from 1st AI, 2nd set of identified features from second AI, or3rd set of identified features from 3rd AI), corresponding feature tagsand corresponding feature probabilities, a component 1669 configured toidentify consensus features, a component 1670 configured to determine anoverall confidence level for each consensus feature, a component 1672configured to determine tags to include in a tag list, a component 1674configured to store the tag list in memory, e.g., with a scanned image,a component 1676 configured to communicate a scanned image andassociated tag list to a secure server, a component 1678 configured toreceive search query, e.g., word, from a user, a component 1680configured to conduct a search, e.g., at the appliance or secure server,on a stored tag list to identify stored image(s), and a component 1682configured to respond to a search query including a word matching a tagassociated with a scanned image by providing the scanned image, e.g., toa display or the device from which the query was received.

Assembly of components 1600 further includes a component 1700 configuredto determine an overall confidence level for an individual consensusfeature. Component 1700 includes a component 1704 configured to generatea probability of correct recognition of a feature by utilized AIsystems, a component 1706 configured to compute a posterior densityprobability of a feature being in an image using confidence levelindicated by a 1st AI system and an average probability as prior, acomponent 1708 configured to compute a posterior density probability ofa feature being in an image using confidence level indicated by a 2nd AIsystem and previous probability generated based on 1st AI confidence asprior, a component 1710 configured to compute a posterior densityprobability of a feature being in an image using confidence levelindicated by a 3rd AI system and previous probability generated based on2nd AI confidence as prior.

Assembly of components 1600 further includes a component 1800 configuredto determine tags to include in a tag list associated with a scannedimage. Component 1800 includes a component 1802 configured to receiveand overall confidence level for a consensus feature and a tagassociated with a scanned image, a component 1804 configured todetermine if the overall confidence level is below a minimum thresholdand to control operation as a function of the determination, a component1813 configured to discard the tag/feature in response to adetermination that the overall confidence level is below the minimumconfidence threshold level, and a component 1806 configured to determineif the overall confidence level is over an upper confidence levelthreshold and to control operation as a function of the determination.Assembly of components 1800 further includes a component 1808 configuredto present a tag to a user as a suggested tag for a scanned image raterequest confirmation that the tag should be used in response to adetermination that the overall confidence level is not over the upperconfidence level threshold but is greater than or equal to the minimumconfidence threshold level, a component 1814 configured to designate thetag corresponding to a feature as being a confirmed tag in response to adetermination that the overall confidence level is over the upperconfidence level threshold, a component 1810 configured to receive userresponse to a tag suggestion, a component 1812 configured to determineif the user confirmed that the tag should be used as a tag for thescanned image and to control operation as a function of thedetermination, a component 18131 configured to discard the tag/featurein response to a determination that the user did not confirm that hatthe tag should be used as tag for the scanned image, a component 18141configured to designate the tag and corresponding feature as beingconfirmed in response to a determination that the user confirmed thatthe tag should be used as a tag for the scanned image.

Component 1800 further includes a component 1816 configured to identifysynonym(s) to a confirmed tag, and a component 1818 configured to add aconfirmed tag and identified synonym(s) if any to a confirmed tag listcorresponding to the scanned image.

In the following numbered lists, numbered embodiments refer toembodiments in the same list as the claim referring to another, e.g.,preceding, numbered embodiment.

First Numbered List of Exemplary Method Embodiments Method Embodiment 1

A method comprising: processing (1256) a scanned image using a firstartificial intelligence (AI) system to detect features in the scannedimage corresponding to word tags and to generate correspondingprobabilities that the detected features are present in the scannedimage; processing (1258) the scanned image using a second artificialintelligence (AI) system to detect features in the scanned imagecorresponding to word tags and to generate corresponding probabilitiesthat the detected features are present in the scanned image; identifying(1258) consensus features, consensus features being features detected bymultiple AI systems which processed said scanned image; determining(1270), for at least a first identified consensus feature, an overallprobability (e.g., overall confidence level), based on a firstprobability generated by the first AI system for the first consensusfeature and on a second probability generated by the second AI systemfor the first consensus feature; and determining (1272) whether or notto include a tag corresponding to the first consensus feature in a taglist corresponding to the scanned image.

Method Embodiment 2

The method of Method Embodiment 1, wherein determining (1270), for atleast a first identified consensus feature, the overall probabilityincludes: computing (1306) a first posterior density probability usingas a first input a probability value generated based on an averagedetection rate with respect to identification of features associated thefirst artificial intelligence (AI) system and a probability valuegenerated based on an average detection rate with respect toidentification of features associated the second artificial intelligence(AI) system and using the probability generated by the first AI systemfor the first consensus feature as a second input.

Method Embodiment 3

The method of Method Embodiment 2, wherein determining (1270), for atleast a first identified consensus feature, the overall probabilityfurther includes: computing (1308) a second posterior densityprobability using as a first input the first posterior densityprobability and using the probability generated by the second AI systemfor the first consensus feature as a second input.

Method Embodiment 4

The method of Method Embodiment 3, wherein determining (1270), for atleast a first identified consensus feature, the overall probabilityfurther includes: computing (1310) a third posterior density probabilityusing as a first input the second posterior density probability andusing the probability generated by the third AI system for the firstconsensus feature as a second input.

Method Embodiment 5

The method of Method Embodiment 4, further comprising: using said thirdposterior density as the overall probability for the first consensusfeature.

Method Embodiment 6

The method of Method Embodiment 5, further comprising: storing (1274)the tag list associated with the scanned image; and using (1278) thestored tag list to determine if a word in a search corresponds to thescanned image.

Method Embodiment 7

The method of Method Embodiment 6, further compressing: responding(1278) to a search by providing the scanned image to a display anddisplaying the image.

Second Numbered List of Exemplary Method Embodiments Method Embodiment 1

A method of processing one or more images, the method comprising:processing (1256) (e.g., at a first feature detection system such as afirst network based AI system (97) or performing the processing at thehome appliance (108) using a first feature detection module) a scannedimage using a first artificial intelligence (AI) system (97) to detectfeatures in the scanned image corresponding to word tags and to generatecorresponding probabilities that the detected features are present inthe scanned image; processing (1258) (e.g., at a second featuredetection system such as a second network based AI system (98) orperforming the processing at the home appliance using a second featuredetection module) the scanned image using a second artificialintelligence (AI) system (98) to detect features in the scanned imagecorresponding to word tags and to generate corresponding probabilitiesthat the detected features are present in the scanned image; identifying(1269) (e.g. at a customer premises device such as customer/homemanagement appliance (108) or secure server (102)) consensus features inthe scanned image, consensus features being features detected bymultiple AI systems (97, 98 and/or 99) which processed said scannedimage (where identifying (1269) consensus features in some embodimentsincludes examining a feature, e.g., a first feature identified by thefirst AI system to determine if the same feature was also identified bythe second and/or third AI system with the first feature beingidentified as a consensus feature in response to determining that it wasidentified by two and in some but not all cases 3 of the AI systems);determining (1270), (e.g. at a customer premises device such ascustomer/home management appliance (108) or secure server (102)) for atleast a first identified consensus feature, an overall probability(e.g., overall confidence level), based on a first probability generatedby the first AI system (97) for the first consensus feature and on asecond probability generated by the second AI system (98) for the firstconsensus feature (e.g., in some embodiments the overall confidencelevel for the first identified consensus features is a weighted sum ofthe probability and/or confidence level indicated by each AI systemwhich identified the feature. However in other embodiments the overallprobability is generated from one or more posterior densityprobabilities); and determining (1272) (e.g. at a customer premisesdevice such as customer/home management appliance (108) or secure server(102)) whether or not to include a tag corresponding to the firstconsensus feature in a tag list corresponding to the scanned image.

Method Embodiment 1A

The method of Method Embodiment 1, wherein said determining an overallprobability includes generating an average probability for a firstfeature identified by multiple AI systems by summing probabilitiesindicated by each AI system that the feature was present in and dividingby the number of probabilities that are combined, —for Example if AIsystem 1 returned probability P1 for feature 1 and AI system 2 returnedprobability P2 for feature 1 and AI system 3 returned probability P3 forfeature 1 the overall probability for feature 1 can be determined by theequation (P1+P2+P3)/3. In some cases the probabilities P1, P2 and P3 maybe weighted prior to combining, e.g., by weights that may bepredetermined and reflect the known reliability of various AI systemsused to detect features. In some cases the weight corresponding to an AIsystem is based on the average detection rate of the AI system or basedon average detection rates of the AI system being used. In one suchembodiment the overall probability for feature 1 is determined by theequation overall probability of feature 1—(W1P1+W2P2+W3P3)/3 where W1 isa weight indicative of the reliability of the AI system 1 which detectedfeature 1 and provided probability P1 as a confidence level of detectingfeature 1, where W2 is a weight indicative of the reliability of the AIsystem 2 which detected feature 1 and provided probability P2 as aconfidence level of detecting feature 1, and where W3 is a weightindicative of the reliability of the AI system 3 which detected feature1 and provided probability P3 as a confidence level of detecting feature1).

Method Embodiment 1B

The method of Method Embodiment 1 or 1A, wherein determining (1272)(e.g. at a customer premises device such as customer/home managementappliance (108) or secure server (102)) whether or not to include a tagcorresponding to the first consensus feature in a tag list correspondingto the scanned image includes determining if the overall probability forthe first consensus features exceeds a threshold; including the tagcorresponding to the first consensus feature in the tag list when theoverall probability for the first consensus feature exceeds thethreshold; and not including the tag corresponding to the firstconsensus feature in the tag list when the overall probability for thefirst consensus feature is equal to or below the threshold.

Method Embodiment 1C

The method of Method Embodiment 1B, wherein said threshold in someembodiments is a predetermined probability threshold (e.g. a 70%probability threshold) of the feature corresponding to the thresholdbeing present in the first image.

Method Embodiment 2

The method of Method Embodiment 1, wherein determining (1270) (e.g. at acustomer premises device such as customer/home management appliance(108) or secure server (102)) for at least a first identified consensusfeature, the overall probability includes: computing (1306) (e.g. at acustomer premises device such as customer/home management appliance(108) or secure server (102)) a first posterior density probabilityusing as a first input a probability value generated based on an averagedetection rate with respect to identification of features associated thefirst artificial intelligence (AI) system (97) and a probability valuegenerated based on an average detection rate with respect toidentification of features associated the second artificial intelligence(AI) system (98) and using the probability generated by the first AIsystem (97) for the first consensus feature as a second input.

Method Embodiment 3

The method of Method Embodiment 2, wherein determining (1270) (e.g. at acustomer premises device such as customer/home management appliance(108) or secure server (102)), for at least a first identified consensusfeature, the overall probability further includes: computing (1308))(e.g. at a customer premises device such as customer/home managementappliance (108) or secure server (102)), a second posterior densityprobability using as a first input the first posterior densityprobability and using the probability generated by the second AI system(98) for the first consensus feature as a second input.

Method Embodiment 4

The method of Method Embodiment 3, wherein determining (1270) (e.g. at acustomer premises device such as customer/home management appliance(108) or secure server (102, for at least a first identified consensusfeature, the overall probability further includes: computing (1310)(e.g. at a customer premises device such as customer/home managementappliance (108) or secure server (102)) a third posterior densityprobability using as a first input the second posterior densityprobability and using the probability generated by the third AI system(99) for the first consensus feature as a second input.

Method Embodiment 5

The method of Method Embodiment 4, further comprising: using (e.g. at acustomer premises device such as customer/home management appliance(108) or secure server (102) said third posterior density as the overallprobability for the first consensus feature.

Method Embodiment 5AA

The method of Method Embodiment 1, wherein the first (97) and second(98) artificial intelligence (AI) systems (e.g. an Internet accessibleGoogle image feature recognition system, an Internet accessibleMicrosoft image feature recognition system and a third party providersystem) are different Internet based systems, the method furthercomprising: sending (1255), from the home appliance (108) the firstscanned image to the first AI (97) and second AI systems (98) via anInternet communications network (104).

Method Embodiment 5A

The method of Method Embodiment 1, wherein the first AI system (97) is acomputer system that was trained on a first set of training dataincluding a first set of images, images in said first set of imagesincluding one or more known features, a tag being associated with eachknown feature.

Method Embodiment 5B

The method of Method Embodiment 5A, wherein the first AI system (97) isa first cloud based system, the method further comprising: communicating(1255) the scanned image via a communications network ((e.g.,potentially via secure server (102)) from a customer premise device(e.g., customer/home management appliance (108)) which receives imagesand provides in response to each received images one or more tags with aconfidence level to the first AI system (97).

Method Embodiment 5C

The method of Method Embodiment 5A, further comprising: receiving(1268′) information indicating features identified by the first AIsystem with corresponding tags and confidence levels (e.g., for eachfeature that was identified, the feature is indicated in a list, acorresponding confidence metric, e.g., detection probability is providedin the list, and one or more tags corresponding to the feature areprovided in the list that is received); receiving (1268″) informationindicating features identified by the second AI system withcorresponding tags and confidence levels (e.g., for each feature thatwas identified by the second AI system, the feature is indicated in alist received from the second AI system with a corresponding confidencemetric (e.g., detection probability is provided in the list), and one ormore tags corresponding to the feature are provided in the list that isreceived from the second AI system); and receiving (1268′″) informationindicating features identified by the third AI system with correspondingtags and confidence levels (e.g., for each feature that was identifiedby the third AI system, the feature is indicated in a list received fromthe third AI system with a corresponding confidence metric (e.g.,detection probability is provided in the list), and one or more tagscorresponding to the feature are provided in the list that is receivedfrom the third AI system).

Method Embodiment 6

The method of Method Embodiment 5, further comprising: storing (1274)the tag list associated with the scanned image (e.g., in the secureserver 102 or memory of the home appliance 108).

Method Embodiment 7

The method of Method Embodiment 6, further comprising: receiving (1278)a query from a user (e.g., user of home appliance (108)); and using(1280) the stored tag list to determine if a word in a searchcorresponds to the scanned image.

Method Embodiment 8

The method of Method Embodiment 7, further compressing: responding(1282) to a search by providing the scanned image to a display anddisplaying the image.

Method Embodiment 9

The method of Method Embodiment 1, wherein the first (97) and second(98) artificial intelligence (AI) systems are different Internet basedsystems, the method further comprising: sending (1255), from the homeappliance (108) the first scanned image to the first AI (97) and secondAI systems (98) via an Internet communications network (104).

First Numbered List of Exemplary Apparatus Embodiments ApparatusEmbodiment 1

A an apparatus (300 or 900) for processing one or more images, theapparatus comprising: a receiver (356, 360, 914 or 918) configured toreceive: i) first information indicating a first list of featuresdetected by a first feature detection system (97) to be present in ascanned image and corresponding probabilities that the detected featuresare present in the scanned image and ii) second information indicating asecond list of features detected by a second feature detection system(98) to be present in the scanned image; memory (302 or 912) storingsaid first and second information (323) and said scanned image (327);and a processor (306 or 906), configured to control the apparatus (300or 108) to: identify (1269) (e.g. at a customer premises device such ascustomer/home management appliance (108) or secure server (102))consensus features in the scanned image, consensus features beingfeatures detected by multiple AI systems (97, 98 and/or 99) whichprocessed said scanned image (where identifying (1269) consensusfeatures in some embodiments includes examining a feature, e.g., a firstfeature identified by the first AI system to determine if the samefeature was also identified by the second and/or third AI system withthe first feature being identified as a consensus feature in response todetermining that it was identified by two and in some but not all cases3 of the AI systems); determine (1270), (e.g. at a customer premisesdevice such as customer/home management appliance (108) or secure server(102)) for at least a first identified consensus feature, an overallprobability (e.g., overall confidence level), based on a firstprobability generated by the first AI system (97) for the firstconsensus feature and on a second probability generated by the second AIsystem (98) for the first consensus feature (e.g., in some embodimentsthe overall confidence level for the first identified consensus featuresis a weighted sum of the probability and/or confidence level indicatedby each AI system which identified the feature. However in otherembodiments the overall probability is generated from one or moreposterior density probabilities); and determine (1272) (e.g. at acustomer premises device such as customer/home management appliance(108) or secure server (102)) whether or not to include a tagcorresponding to the first consensus feature in a tag list correspondingto the scanned image.

Apparatus Embodiment 1A

The apparatus (300 or 900) of Apparatus Embodiment 1, wherein saidprocessor (306 or 906) is configured, as part of determining an overallprobability to: generate an average probability for a first featureidentified by multiple AI systems by summing probabilities indicated byeach AI system that the feature was present in and dividing by thenumber of probabilities that are combined, —for Example if AI system 1returned probability P1 for feature 1 and AI system 2 returnedprobability P2 for feature 1 and AI system 3 returned probability P3 forfeature 1 the overall probability for feature 1 can be determined by theequation (P1+P2+P3)/3. In some cases the probabilities P1, P2 and P3 maybe weighted prior to combining, e.g., by weights that may bepredetermined and reflect the known reliability of various AI systemsused to detect features. In some cases the weight corresponding to an AIsystem is based on the average detection rate of the AI system or basedon average detection rates of the AI system being used. In one suchembodiment the overall probability for feature 1 is determined by theequation overall probability of feature 1—(W1P1+W2P2+W3P3)/3 where W1 isa weight indicative of the reliability of the AI system 1 which detectedfeature 1 and provided probability P1 as a confidence level of detectingfeature 1, where W2 is a weight indicative of the reliability of the AIsystem 2 which detected feature 1 and provided probability P2 as aconfidence level of detecting feature 1, and where W3 is a weightindicative of the reliability of the AI system 3 which detected feature1 and provided probability P3 as a confidence level of detecting feature1).

Apparatus Embodiment 1B

The apparatus (300 or 900) of Apparatus Embodiment 1 or 1A, wherein theprocessor (306 or 906) is configured to control the apparatus to:determine (1272) (e.g. at a customer premises device such ascustomer/home management appliance (108) or secure server (102)) whetheror not to include a tag corresponding to the first consensus feature ina tag list corresponding to the scanned image includes determining ifthe overall probability for the first consensus features exceeds athreshold; including the tag corresponding to the first consensusfeature in the tag list when the overall probability for the firstconsensus feature exceeds the threshold; and not including the tagcorresponding to the first consensus feature in the tag list when theoverall probability for the first consensus feature is equal to or belowthe threshold.

Apparatus Embodiment 1C

The apparatus (300 or 900) of Apparatus Embodiment 1B, wherein saidthreshold in some embodiments is a predetermined probability threshold(e.g. a 70% probability threshold) of the feature corresponding to thethreshold being present in the first image.

Apparatus Embodiment 2

The apparatus (300 or 900) of Apparatus Embodiment 1, wherein theprocessor (306 or 906) is configured, as part of determining (1270)(e.g. at a customer premises device such as customer/home managementappliance (108) or secure server (102)) for at least a first identifiedconsensus feature, the overall probability to control the apparatus to:compute (1306) (e.g. at a customer premises device such as customer/homemanagement appliance (108) or secure server (102)) a first posteriordensity probability using as a first input a probability value generatedbased on an average detection rate with respect to identification offeatures associated the first artificial intelligence (AI) system (97)and a probability value generated based on an average detection ratewith respect to identification of features associated the secondartificial intelligence (AI) system (98) and using the probabilitygenerated by the first AI system (97) for the first consensus feature asa second input.

Apparatus Embodiment 3

The apparatus (300 or 900) of Apparatus Embodiment 2, wherein theprocessor (306 or 906) is further configured, as part of controlling theapparatus to determine (1270) (e.g. at a customer premises device suchas customer/home management appliance (108) or secure server (102)), forat least a first identified consensus feature, the overall probabilityto control the apparatus to: compute (1308)) (e.g. at a customerpremises device such as customer/home management appliance (108) orsecure server (102)), a second posterior density probability using as afirst input the first posterior density probability and using theprobability generated by the second AI system (98) for the firstconsensus feature as a second input.

Apparatus Embodiment 4

The apparatus (300 or 900) of Apparatus Embodiment 3, wherein theprocessor (306 or 906) is further configured, to control the apparatusas part of determining (1270) (e.g. at a customer premises device suchas customer/home management appliance (108) or secure server (102, forat least a first identified consensus feature, the overall probabilityto: compute (1310) (e.g. at a customer premises device such ascustomer/home management appliance (108) or secure server (102)) a thirdposterior density probability using as a first input the secondposterior density probability and using the probability generated by thethird AI system (99) for the first consensus feature as a second input.

Apparatus Embodiment 5

The apparatus (300) of Apparatus Embodiment 4, further comprising:operating the processor (306) to control the apparatus (300) to: use(e.g. at a customer premises device such as customer/home managementappliance (108) or secure server (102)) said third posterior density asthe overall probability for the first consensus feature.

Apparatus Embodiment 5AA

The apparatus (300) of Apparatus Embodiment 1, wherein the first (97)and second (98) feature detection systems (e.g. an Internet accessibleGoogle image feature recognition system, an Internet accessibleMicrosoft image feature recognition system and a third party providersystem) are different Internet based systems, the apparatus furthercomprising: a transmitter (358 or 362) for sending (1255), from the homeappliance (108) the first scanned image to the first AI (97) and secondAI systems (98) via an Internet communications network (104).

Apparatus Embodiment 5A

The apparatus (300 or 900) of Apparatus Embodiment 1, wherein the firstfeature recognition system (97) is a computer system that was trained ona first set of training data including a first set of images, images insaid first set of images including one or more known features, a tagbeing associated with each known feature.

Apparatus Embodiment 5B

The apparatus (300) of Apparatus Embodiment 5A, wherein the firstfeature recognition system (97) is a first cloud based system; andwherein the processor (306) is further configured to control theapparatus (300) to: communicate (1255) the scanned image via acommunications network ((e.g., potentially via secure server (102)) froma customer premise device (e.g., customer/home management appliance(108)) which receives images and provides in response to each receivedimages one or more tags with a confidence level to the first AI system(97).

Method Embodiment 5C

The apparatus (300) of Apparatus Embodiment 5A, wherein the receiver(356 or 360) is further configured to: receive (1268′) informationindicating features identified by the first feature recognition systemwith corresponding tags and confidence levels (e.g., for each featurethat was identified, the feature is indicated in a list, a correspondingconfidence metric, e.g., detection probability is provided in the list,and one or more tags corresponding to the feature are provided in thelist that is received); receive (1268″) information indicating featuresidentified by the second feature recognition system with correspondingtags and confidence levels (e.g., for each feature that was identifiedby the second AI system, the feature is indicated in a list receivedfrom the second AI system with a corresponding confidence metric (e.g.,detection probability is provided in the list), and one or more tagscorresponding to the feature are provided in the list that is receivedfrom the second AI system); and receive (1268′″) information indicatingfeatures identified by the third feature recognition system withcorresponding tags and confidence levels (e.g., for each feature thatwas identified by the third AI system, the feature is indicated in alist received from the third AI system with a corresponding confidencemetric (e.g., detection probability is provided in the list), and one ormore tags corresponding to the feature are provided in the list that isreceived from the third AI system).

Apparatus Embodiment 6

The apparatus (300) of Apparatus Embodiment 5, wherein the memory (302)stores (1274) the tag list (325) associated with the scanned image(e.g., in the secure server 102 or memory of the home appliance 108).

Apparatus Embodiment 7

The apparatus (300) of Apparatus Embodiment 6, further comprising: auser input device (310) configured to receive (1278) a query from a user(e.g., user of home appliance (108)); and wherein the processor isfurther configured to: use (1280) the stored tag list (325) to determineif a word in a search corresponds to the scanned image.

Apparatus Embodiment 8

The apparatus (300) of Apparatus Embodiment 7, wherein the processor isfurther configured to control the apparatus (300 or 108) to: respond(1282) to a search by providing the scanned image to a display (308) anddisplay the image.

Third Numbered List of Method Embodiments Method Embodiment 1

A method of operating a customer premises device (e.g., home appliances108 or 300) to associate tags with a scanned image, the methodcomprising: i) generating (1270)) a confidence level indicating a levelof confidence with respect to a first feature being in a scanned image(e.g., generate an overall confidence value which is the confidencelevel based on received confidence values from multiple AI systems whichdetected a feature in the scanned image) or ii) receiving (1402) (e.g.,from a network based feature recognition system 97, 98 or 99 in whichcase a single confidence value received from a single AI system is usedas an overall confidence level instead of an overall confidence valuegenerated from the output of multiple feature recognition systems) theconfidence level indicating a level of confidence with respect to afirst feature being in a scanned image; determining (1406) if theconfidence level is over an upper confidence level threshold; andassociating (1418) a word corresponding to the first feature with thescanned image as a first tag when the confidence level is determined tobe over the upper confidence level threshold.

Method Embodiment 2

The method of Method Embodiment 1, further comprising: presenting (1408)the word to a user as a suggested tag for the scanned image when theconfidence level is below the upper confidence level threshold and abovea minimum threshold level.

Method Embodiment 3

The method of Method Embodiment 2, further comprising: receiving (1410)a user response to the tag suggestion; and i) discarding (1413) thesuggested tag when the user does not confirm that the tag should be usedfor the scanned image; and ii) designating (1414) the tag as a confirmedtag when user confirms that the tag should be used and associating(1418) the word to be used as a tag with the scanned image.

Method Embodiment 4

The method of Method Embodiment 3, further comprising: identifying(1416) one or more synonyms corresponding to a user confirmed tag or tagautomatically associated with the scanned image; and associating (1418)the identified one or more synonyms with the scanned image.

Method Embodiment 5

The method of Method Embodiment 4, further comprising: receiving (1402)a second confidence level indicating a level of confidence with respectto an additional feature being in the scanned image; determining (1406)if the second confidence level is over the upper confidence levelthreshold; and associating (1418) a word corresponding to the additionalfeature with the image as a tag when the confidence level is determinedto be over the upper confidence level threshold.

Method Embodiment 6

The method of Method Embodiment 5, further comprising: wherein theconfidence level is an overall confidence level generated fromconfidence levels returned by multiple artificial intelligence systems(e.g., neural network based feature recognition systems).

Second Numbered List of Exemplary Apparatus Embodiments ApparatusEmbodiment 1

An apparatus (300 or 108) comprising: a receiver (356); memory (302)storing a scanned image; and a processor (306) configured to control theapparatus to: i) generate (1270)) a confidence level indicating a levelof confidence with respect to a first feature being in a scanned image(e.g., generate an overall confidence value which is the confidencelevel based on received confidence values from multiple AI systems whichdetected a feature in the scanned image) or ii) receive (1402) (e.g.,from a network based feature recognition system 97, 98 or 99 in whichcase a single confidence value received from a single AI system is usedas an overall confidence level instead of an overall confidence valuegenerated from the output of multiple feature recognition systems) theconfidence level indicating a level of confidence with respect to afirst feature being in a scanned image; determine (1406) if theconfidence level is over an upper confidence level threshold; andassociate (1418), in the memory (302), a word corresponding to the firstfeature with the scanned image as a first tag when the confidence levelis determined to be over the upper confidence level threshold.

Apparatus Embodiment 2

The apparatus (300 or 108) of Apparatus Embodiment 1, wherein theprocessor (306) is further configured to control the apparatus to:present (1408) the word to a user as a suggested tag for the scannedimage when the confidence level is below the upper confidence levelthreshold and above a minimum threshold level.

Apparatus Embodiment 3

The apparatus (300 or 108) of Apparatus Embodiment 2, wherein theprocessor (306) is further configured to control the apparatus (300 or108) to: receive (1410) a user response to the tag suggestion; and i)discard (1413) the suggested tag when the user does not confirm that thetag should be used for the scanned image and ii) designate (1414) thetag as a confirmed tag when user confirms that the tag should be usedand associating (1418) the word to be used as a tag with the scannedimage.

Apparatus Embodiment 4

The apparatus (300 or 108) of Apparatus Embodiment 3, wherein theprocessor is further configured to control the apparatus to: identify(1416) one or more synonyms corresponding to a user confirmed tag or tagautomatically associated with the scanned image; and associate (1418)the identified one or more synonyms with the scanned image.

Apparatus Embodiment 5

The apparatus (300 or 108) of Apparatus Embodiment 4, wherein theprocessor (306) is further configured to control the apparatus (300 or108) to: receive (1402) a second confidence level indicating a level ofconfidence with respect to an additional feature being in the scannedimage; determine (1406) if the second confidence level is over the upperconfidence level threshold; and associate (1418) a word corresponding tothe additional feature with the image as a tag when the confidence levelis determined to be over the upper confidence level threshold.

Apparatus Embodiment 6

The apparatus (300 or 108) of Apparatus Embodiment 5, furthercomprising: wherein the confidence level is an overall confidence levelgenerated from confidence levels returned by multiple artificialintelligence systems (e.g., neural network based feature recognitionsystems).

The methods of various embodiments may be implemented using software,hardware and/or a combination of software and hardware. In someembodiments, modules are implemented as physical modules. In some suchembodiments, the individual physical modules are implemented inhardware, e.g., as circuits, or include hardware, e.g., circuits, withsome software. In other embodiments, the modules are implemented assoftware modules which are stored in memory and executed by a processor,e.g., general purpose computer. Various embodiments are also directed tomachine, e.g., computer, readable medium, e.g., ROM, RAM, CDs, harddiscs, etc., which include machine readable instructions for controllinga machine, e.g., processor or computer system, to implement one or moresteps of a method.

It is understood that the specific order or hierarchy of steps in theprocesses disclosed is an example of exemplary approaches. Based upondesign preferences, it is understood that the specific order orhierarchy of steps in the processes may be rearranged to some extentwhile remaining within the scope of the present disclosure.

In various embodiments nodes described herein are implemented using oneor more modules to perform the steps corresponding to one or moremethods. Thus, in some embodiments various features are implementedusing modules. Such modules may be implemented using software, hardwareor a combination of software and hardware. Many of the above describedmethods or method steps can be implemented using machine executableinstructions, such as software, included in a non-transitory machinereadable medium such as a memory device, e.g., RAM, floppy disk, etc. tocontrol a machine, e.g., general purpose computer with or withoutadditional hardware, to implement all or portions of the above describedmethods, e.g., in one or more nodes. Accordingly, among other things,various embodiments are directed to a machine, e.g., computer, readablemedium including machine, e.g., computer, executable instructions forcausing a machine, e.g., computer, processor and/or associated hardware,to perform one or more of the steps of the above-described method(s).

Some embodiments are directed to a computer program product comprising acomputer-readable medium comprising code for causing a computer, ormultiple computers, to implement various functions, steps, acts and/oroperations, e.g. one or more steps described above. Depending on theembodiment, the computer program product can, and sometimes does,include different code for each step to be performed. Thus, the computerprogram product may, and sometimes does, include code for eachindividual step of the method or methods described herein. The code maybe in the form of machine, e.g., computer, executable instructionsstored on a computer-readable medium such as a RAM (Random AccessMemory), ROM (Read Only Memory) or other type of storage device. Inaddition to being directed to a computer program product, someembodiments are directed to a processor configured to implement one ormore of the various functions, steps, acts and/or operations of one ormore methods described above. Accordingly, some embodiments are directedto a processor, e.g., CPU, configured to implement some or all of thesteps of the methods described herein. The processor may be for use in,e.g., a communications device or other device described in the presentapplication.

Numerous additional variations on the methods and apparatus of thevarious embodiments described above will be apparent to those skilled inthe art in view of the above description. Such variations are to beconsidered within the scope of the invention.

What is claimed is:
 1. A method of operating a customer premises device(e.g., home appliances 108 or 300) to associate tags with a scannedimage, the method comprising: i) generating (1270)) a confidence levelindicating a level of confidence with respect to a first feature beingin a scanned image (e.g., generate an overall confidence value which isthe confidence level based on received confidence values from multipleAI systems which detected a feature in the scanned image) or ii)receiving (1402) the confidence level indicating a level of confidencewith respect to a first feature being in a scanned image; determining(1406) if the confidence level is over an upper confidence levelthreshold; and associating (1418) a word corresponding to the firstfeature with the scanned image as a first tag when the confidence levelis determined to be over the upper confidence level threshold.
 2. Themethod of claim 1, further comprising: presenting (1408) the word to auser as a suggested tag for the scanned image when the confidence levelis below the upper confidence level threshold and above a minimumthreshold level.
 3. The method of claim 2, further comprising: receiving(1410) a user response to the tag suggestion; and i) discarding (1413)the suggested tag when the user does not confirm that the tag should beused for the scanned image; and ii) designating (1414) the tag as aconfirmed tag when user confirms that the tag should be used andassociating (1418) the word to be used as a tag with the scanned image.4. The method of claim 3, further comprising: identifying (1416) one ormore synonyms corresponding to a user confirmed tag or tag automaticallyassociated with the scanned image; and associating (1418) the identifiedone or more synonyms with the scanned image.
 5. The method of claim 4,further comprising: receiving (1402) a second confidence levelindicating a level of confidence with respect to an additional featurebeing in the scanned image; determining (1406) if the second confidencelevel is over the upper confidence level threshold; and associating(1418) a word corresponding to the additional feature with the image asa tag when the confidence level is determined to be over the upperconfidence level threshold.
 6. The method of claim 5, furthercomprising: wherein the confidence level is an overall confidence levelgenerated from confidence levels returned by multiple artificialintelligence systems (e.g., neural network based feature recognitionsystems).
 7. An apparatus (300 or 108) comprising: a receiver (356);memory (302) storing a scanned image; and a processor (306) configuredto control the apparatus to: i) generate (1270)) a confidence levelindicating a level of confidence with respect to a first feature beingin a scanned or ii) receive (1402) the confidence level indicating alevel of confidence with respect to a first feature being in a scannedimage; determine (1406) if the confidence level is over an upperconfidence level threshold; and associate (1418), in the memory (302), aword corresponding to the first feature with the scanned image as afirst tag when the confidence level is determined to be over the upperconfidence level threshold.
 8. The apparatus (300 or 108) of claim 7,wherein the processor (306) is further configured to control theapparatus to: present (1408) the word to a user as a suggested tag forthe scanned image when the confidence level is below the upperconfidence level threshold and above a minimum threshold level.
 9. Theapparatus (300 or 108) of claim 7, wherein the processor (306) isfurther configured to control the apparatus (300 or 108) to: receive(1410) a user response to the tag suggestion; and i) discard (1413) thesuggested tag when the user does not confirm that the tag should be usedfor the scanned image and ii) designate (1414) the tag as a confirmedtag when user confirms that the tag should be used and associating(1418) the word to be used as a tag with the scanned image.
 10. Theapparatus (300 or 108) of claim 9, wherein the processor is furtherconfigured to control the apparatus to: identify (1416) one or moresynonyms corresponding to a user confirmed tag or tag automaticallyassociated with the scanned image; and associate (1418) the identifiedone or more synonyms with the scanned image.
 11. The apparatus (300 or108) of claim 10, wherein the processor (306) is further configured tocontrol the apparatus (300 or 108) to: receive (1402) a secondconfidence level indicating a level of confidence with respect to anadditional feature being in the scanned image; determine (1406) if thesecond confidence level is over the upper confidence level threshold;and associate (1418) a word corresponding to the additional feature withthe image as a tag when the confidence level is determined to be overthe upper confidence level threshold.
 12. The apparatus (300 or 108) ofclaim 11, further comprising: wherein the confidence level is an overallconfidence level generated from confidence levels returned by multipleartificial intelligence systems.